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Executive  Summary 


Despite  a  rich  and  extensive  history  of  research  aimed  at  understanding  the  complexities  of 
intelligence  analysis,  the  processes  used  to  turn  data  into  actionable  intelligence  information,  and 
the  skills  and  abilities  needed  to  perform  analysis-related  activities,  relatively  little  is  known 
about  what  makes  an  analyst  successful.  The  increasing  importance  and  complexity  of 
intelligence  in  support  of  Army  missions  underlies  a  growing  need  to  increase  analysts’ 
reliability  and  performance  on  complex,  analytical  decision-making  tasks.  To  meet  this  need,  the 
Army  must  develop  a  selection  process  that  assures  that  prospective  analysts  have  the  required 
characteristics  to  meet  the  job  demands. 

One  possible  means  to  develop  this  selection  process  is  to  identify  personnel  characteristics  that 
contribute  to  performance  on  mission-critical  intelligence-related  tasks,  describe  the  impacts 
these  characteristics  have  on  overall  mission  performance,  and  then  identify  ways  to  increase  the 
degree  to  which  intelligence  analysts  possess  these  attributes. 

The  focus  of  this  project  was  to  begin  providing  the  needed  selection  mechanism.  Our  goals 
were  to  identify  the  cognitive  attributes  required  for  effective  intelligence  analysis  and  quantify 
the  relationship  of  individual  differences  in  those  attributes  and  system-level  performance.  Our 
approach  included  two  primary  efforts:  (1)  a  review  of  the  scientific  literature  and  research  reports 
and  (2)  development  of  a  task  network  model  of  the  all-source  intelligence  analyst  function. 

Rather  than  focusing  on  single-source  domains,  such  as  imagery  or  human  intelligence,  where 
specific  cognitive  attributes  may  be  needed  only  for  that  specific  discipline  (e.g.,  visual/spatial 
intelligence  for  imagery  analysts),  the  project  focused  on  the  multisource  analyst.  We  believed 
these  required  attributes  would  be  more  general  to  suit  a  broad  set  of  analytic  activities  within  the 
field  of  Army  intelligence  analysis.  As  such,  the  richest  target  for  this  type  of  analysis  appeared 
to  be  the  35F  Intelligence  Analyst.  The  Military  Occupational  Specialty  (MOS)  35F  Intelligence 
Analyst,  or  all-source  intelligence  analyst,  is  “primarily  responsible  for  supervising,  coordinating 
and  participating  in  the  analysis,  processing  and  distribution  of  strategic  and  tactical 
intelligence.”1 

Having  selected  the  target  MOS,  we  began  by  developing  a  conceptual  framework  based  on  an 
initial  review  of  relevant  literature  that  would  inform  the  search  for  the  required  personnel 
characteristics  and  their  quantitative  relationship  to  task  performance.  Table  ES-1  presents  a 


'u.S.  Army  Intelligence  Analyst,  http://www.goarmy.com/careers-and-jobs/browse-career-and-job-categories/intelligence- 
and-combat-support/intelligence-analyst.html  (accessed  10  September  2011). 


Table  ES-1.  Cognitive  attribute  comparison  approach. 


Core  IP  Function 
(Identify/Store  Information) 


S-T  Memory 
L-T  Memory 
Attention  Management 


Memorization 
Selective  Attention 


Pattern  Recognition 


Time  Sharing 


Mediating  IP  Function 
(Manipulate  Information) 


Situational  Awareness 
Problem  Solving 
Decision  Making 
Adaptability 
Creativity 


Deductive  Reasoning 
Inductive  Reasoning 
Problem  Sensitivity 
Flexibility  of  Closure 
Fluency  of  Ideas 


Originality 


Executive  IP  Function 

(Strategy  Development  &  Use  -  Meta-Cognition 

Involves  Planning,  Executing,  Perseverance/Resilience 

Monitoring) 


Automaticity 


TBD 


comparison  between  a  conceptual  view  based  on  work  presented  in  the  Army  Research 
Institute’s  Science  of  Human  Measures  Workshop ,2  the  Job  Comparison  and  Analysis  Tool 
(JCAT)3  related  work,  and  our  conceptualization  framed  by  the  initial  literature  review  efforts. 

Based  on  this  framework,  we  extended  our  literature  review  to  identify  cognitive  abilities 
(attributes)  required  to  perform  the  35F  IA  job  (as  described  in  section  2.3).  The  attributes 
identified  in  the  literature  review  were  then  validated  by  analyst  subject  matter  experts  at  the 
Intelligence  Center  of  Excellence  at  Fort  Huachuca,  AZ. 

This  information  was  then  fed  into  a  second-stage  literature  review  aimed  at  identifying 
quantitative  links  between  intelligence-related  tasks  and  the  specific  attributes  identified.  The 
second-stage  literature  review  began  by  identifying  existing  measures  of  the  target  attributes  and 
using  those  measures  to  search  for  empirical  studies  that  reported  significant  quantitative 
relationships  between  the  measured  attributes  and  performance  on  a  task  that  is  in  some  way 
analogous  to  the  analyst  tasks.  Along  with  the  measure-focused  literature  review,  a  task  network 
model  simulating  35F  performance  in  Stability  and  Support  Operations  and  Full-Spectrum 
Operations  was  created.  The  purpose  of  this  model  was  to  simulate  the  job  of  the  35F  over  a 
1-year  period  and  identify  the  attributes  that  were  most  frequently  used  or  identified  as  most 
important  in  mission  execution.  The  model  outputs  were  used  to  hone  in  on  specific  attributes 
that  could  have  the  greatest  potential  impact  on  overall  mission  performance  for  the  35F. 
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The  outcomes  of  literature  review  and  modeling  efforts  are  a  set  of  actionable  recommendations 
that  could  be  taken  to  improve  the  performance  of  the  MOS  35F  All-Source  Intelligence  Analyst. 
These  recommendations  are  summarized  in  figure  ES-1,  and  detailed  information  on  each 
recommendation  can  be  found  in  section  4. 


Immediate  Term 

IT.l  -  Test  for  Flexibility  of  Closure 

IT.2  -  Validate  critical  thinking  skills  test 
for  IA 

IT.3  -  Training  for  deductive  reasoning 

IT.4  -  Explore  the  role  of  strategy 
development  &  executive  IP 

IT.5  -  Performance  Measure 

Identification 

Mid-Term 

MT.l  -  Impacts  of  team  environment  on 
attributes 

MT.2  -  Further  research  on  ability 
specific  measures  useful  to  selection 

MT.3  -  Experimental  studies  to  identify 
measures  with  discriminant  ability  for 
lAs 

MT.4  -  Explore  non-ability  based  (e.g. 
biographical  information)  measures  for 

IA  selection 

Far  Term 

FT.l  -  Trainability  of  IA  skills 

FT.2  -  Neural  /  EEG  based  measures  for 

IA  abilities 

FT.3  -  Componential  analysis  of  critical 
tasks  to  identify  attribute/skill 
relationships 

Figure  ES-1.  Recommended  follow-on  actions  summary. 


1.  Introduction 


1.1  Objective 

The  United  States  Army  Operating  Concept  2016-2028  (TRADOC  Pam  525-3-1)  (Headquarters, 
2010b)  highlights  the  need  for  a  broad  set  of  intelligence  capabilities  that  are  pushed  down  to 
increasingly  lower  levels  of  the  Army  command  hierarchy.  These  capabilities  include  the  need 
to  combine  traditional  intelligence  with  emerging  sources  of  data,  such  as  civil  affairs  elements 
and  human  terrain  teams.  Included  is  the  need  to  broaden  analysis  beyond  military-focused 
intelligence  to  include  analysis  of  the  complexities  associated  with  political,  military, 
sociological,  infrastructure,  and  informational  aspects  of  the  operating  environment.  The 
intelligence  needs  identified  in  TRADOC  Pam  525-3-1  point  to  the  increasingly  important  role 
that  the  collection,  analysis,  and  dissemination  of  a  broadening  spectrum  of  intelligence 
information  will  have  on  future  military  operations. 

The  need  for  sophisticated  all-source  intelligence  analysts  (IAs)  capable  of  fusing  diverse  sets  of 
information  into  actionable  intelligence  is  a  common  theme  in  the  literature  on  intelligence 
analysis.  Along  with  TRADOC  Pam  525-3-1,  researchers  such  as  Allen  (2008),  Hutchins  et  al. 
(2004;  2007),  Fingar  (2011),  Krizan  (1999),  and  Treverton  and  Gabbard  (2008),  among  others, 
describe  a  complex  environment  characterized  by  vast  amounts  of  information  and  complex, 
changing  relationships  that  must  be  collected,  analyzed,  and  understood  in  order  to  maintain 
intelligence  superiority.  The  increasingly  complex  environment  where  IAs  must  operate 
highlights  the  need  to  reexamine  what  it  takes  to  successfully  perform  this  critical  role  for  Army 
units.  The  goal  of  this  effort  was  to  identify  key  cognitive  attributes  required  to  successfully 
perform  intelligence  analysis  tasks,  demonstrate  a  simulation-based  approach  for  understanding 
the  impact  of  these  attributes  on  overall  performance,  and  develop  a  set  of  actionable 
recommendations  to  improve  analyst  performance  through  selection,  training,  or  capability 
development  activities  in  the  immediate,  near,  and  far  terms. 

1.2  Background 

Despite  a  rich  and  extensive  history  of  research  aimed  at  understanding  the  complexities  of 
intelligence  analysis,  the  processes  used  to  turn  data  into  actionable  intelligence  information  and 
the  skills  and  abilities  needed  to  perform  analysis-related  activities,  relatively  little  is  known 
about  what  makes  an  analyst  successful.  Indeed,  Lowenthal  notes,  “We  have  not  yet  (after  more 
than  55  years)  come  up  with  a  good  picture  in  our  minds — nor  have  we  successfully 
enunciated — just  what  a  professional  intelligence  analyst  ‘looks  like,’  and  how  we  train  and 
develop  this  analyst  across  his  or  her  entire  career — not  just  at  the  outset”  (Swenson,  2003).  This 
lack  of  a  common  understanding  of  the  set  of  core  competencies  (Lewis,  2005)  that  analysts 
must  possess  to  successfully  perform  their  tasks  presents  significant  challenges  to  efforts  aimed 
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at  improving  the  accuracy,  timeliness,  and  usefulness  of  intelligence  products.  The  increasing 
importance  and  complexity  of  intelligence  in  support  of  Army  missions  underlies  a  growing  need 
to  increase  analysts’  reliability  and  performance  on  complex,  analytical  decision-making  tasks. 
To  meet  this  need,  the  Army  must  develop  a  selection  process  that  assures  that  prospective 
analysts  have  the  necessary  knowledge,  skills,  and  abilities  (KSAs)  to  perform  this  complex  task. 

One  possible  means  to  provide  this  selection  process  is  to  identify  personnel  characteristics  that 
contribute  to  performance  on  mission-critical  intelligence-related  tasks,  describe  the  impacts 
these  characteristics  have  on  overall  mission  performance,  and  then  identify  ways  to  increase  the 
degree  to  which  IAs  possess  these  attributes. 

The  focus  of  this  project  was  to  begin  the  process  of  providing  the  needed  selection  mechanism. 
In  order  to  accomplish  this  aim,  the  project  had  the  following  objectives: 

•  Determine  the  critical  tasks  that  all-source  IAs  perform. 

•  Identify  the  personnel  characteristics  (i.e.,  skills  and  attributes)  needed  to  perform  as  well 
as  IAs. 

•  Identify  ways  to  assess  performance  on  those  predictor  skills  and  attributes. 

•  Perform  research  to  quantify  potential  performance  improvements  associated  with  variance 
in  the  identified  attributes. 

•  Use  modeling  to  evaluate  the  impact  on  total  system  performance. 

This  report  describes  the  approach  taken  to  accomplish  these  objectives,  the  results  of  the 
project,  and  a  series  of  recommendations  for  immediate,  near-,  and  far-term  activities  to  improve 
analyst  performance  by  leveraging  the  work  completed  on  this  effort. 

1.3  Approach 

Our  goals  were  to  identify  the  cognitive  attributes  required  for  effective  intelligence  analysis  and 
quantify  the  relationship  of  individual  differences  in  those  attributes  and  system-level 
performance.  Our  approach  included  two  primary  efforts:  (1)  a  review  of  the  scientific  literature 
and  research  reports  and  (2)  development  of  a  task  network  model  of  the  all-source  IA  function. 
The  literature  review  was  used  to  identify  the  cognitive  attributes  that  are  characteristic  of 
successful  analysts  (i.e.,  what  the  successful  analyst  “looks  like”)  and  to  identify  quantitative 
relationships  between  those  cognitive  attributes  and  performance  on  tasks  that  are  similar  to  the 
tasks  of  Army  IAs. 

At  the  outset  of  the  project,  the  team  conducted  an  initial  literature  search  in  the  areas  of 
cognitive  readiness,  decision  making,  general  intelligence  theory,  and  military  and  other 
governmental  intelligence  processes.  A  second  literature  review  was  focused  on  the  cognitive 
attributes  that  had  been  identified  as  important  to  IA  performance  in  previous  research  (Knapp 
and  Tillman,  1998)  and  that  were  confirmed  during  subject  matter  expert  (SME)  interviews. 
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Appendix  A  contains  the  full  set  of  literature  we  reviewed  as  part  of  this  project  effort.  The 
results  of  these  literature  searches  formed  the  basis  for  identifying  the  initial  set  of  cognitive 
attributes  identified  as  predictive  of  successful  performance  in  intelligence  analysis  tasks.  A 
summary  of  the  literature  reviewed  and  the  resulting  list  of  the  key  personnel  attributes  of 
successful  analysts  are  detailed  in  section  2.1  of  this  report. 

Along  with  identifying  the  cognitive  attributes  of  effective  analysts,  we  set  out  to  determine  the 
specific  intelligence  analysis  discipline  that  should  be  the  focus  of  this  effort.  Because  of  the 
variety  of  disciplines  and  activities  that  fall  under  the  umbrella  of  intelligence  analysis  and  the 
corresponding  diversity  of  skill  sets  that  may  be  required  to  perform  those  tasks,  this  effort 
needed  to  be  focused  on  a  set  of  exemplar  tasks  and  skill  requirements  that  would  be  most 
generalizable  across  the  spectrum  of  analysis  domains.  Rather  than  focusing  on  single-source 
domains,  such  as  imagery  (IMINT)  or  human  intelligence  (HUMINT),  where  specific  cognitive 
attributes  may  be  needed  only  for  that  specific  discipline  (e.g.,  visual/spatial  intelligence  for 
imagery  analysts),  the  project  focused  on  the  multisource  analyst.  We  believed  these  required 
attributes  would  be  more  general  to  suit  a  broad  set  of  analytic  activities  within  the  field  of  Army 
intelligence  analysis.  As  such,  the  richest  target  for  this  type  of  analysis  appeared  to  be  the  35F 
IA.  The  Military  Occupational  Specialty  (MOS)  35F  IA,  or  all-source  IA,  is  “primarily 
responsible  for  supervising,  coordinating  and  participating  in  the  analysis,  processing  and 
distribution  of  strategic  and  tactical  intelligence”  (U.S.  Army,  2011).  35F  duties  include  the 
following: 

•  Prepare  all- source  intelligence  products  to  support  the  combat  commander. 

•  Provide  support  to  the  Intelligence,  Surveillance,  and  Reconnaissance  (ISR) 
synchronization  process. 

•  Receive  and  process  incoming  reports  and  messages. 

•  Assist  in  determining  the  significance  and  reliability  of  incoming  information. 

•  Establish  and  maintain  systematic,  cross-reference  intelligence  records  and  files. 

•  Integrate  incoming  information  with  current  intelligence  holdings  and  prepare  and  maintain 
graphics. 

•  Conduct  Intelligence  Preparation  of  the  Battlefield  (IPB)  using  information  from  all 
sources  (U.S.  Army,  2011). 

Because  of  the  broad  nature  of  the  all-source  IA’s  tasks  and  the  requirement  for  the  analyst  to 
examine  a  broad  spectrum  of  data  and  transform  it  into  useful  information  for  battlefield 
intelligence,  the  project  team,  in  conjunction  with  the  Intelligence  Center  of  Excellence,  chose 
the  MOS  35F  as  the  target  of  detailed  analysis  under  this  effort. 
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Part  of  our  approach  focused  on  the  use  of  discrete  event  simulation  as  a  means  of  evaluating 
analyst  performance  employing  an  approach  derived  from  the  human  performance  simulation 
methods  found  in  the  Improved  Performance  Research  Integration  Tool  (IMPRINT).  IMPRINT 
uses  an  embedded  task  taxonomy  to  describe  the  demands  (taxons)  that  task  performance  places 
on  operators  completing  mission-related  tasks.  The  taxons  provide  a  generalizable  method  for 
describing  tasks  in  terms  of  their  component  demands  so  that  environmental  factors  and 
personnel  characteristics  can  influence  task  performance  based  on  defined  performance  shaping 
algorithms  that  are  derived  from  validated  literature  or  experimental  data  collections.  For  this 
effort  we  used  the  IMPRINT  simulation  engine,  Micro  Saint  Sharp,  to  create  a  basic  model  of  the 
tasks  required  to  conduct  activities  related  to  the  high-level  analyst  function  of  “Evaluate  the 
Threat.”  Micro  Saint  Sharp  was  employed  so  that  the  modeling  could  access  more  of  the 
functionality  that  underlies  IMPRINT,  but  it  is  not  accessible  directly  in  the  IMPRINT 
environment.  In  the  model  created  for  this  effort,  we  applied  the  cognitive  attributes  identified 
from  the  literature  search,  in  a  manner  analogous  to  the  resident  taxons  in  IMPRINT,  to  describe 
the  task  demands  placed  on  the  35F.  A  description  of  the  model  is  in  section  2.5  of  this  report. 


2.  Method 


2.1  Cognitive  Attribute  Research 

A  key  component  of  this  effort  was  an  extensive  review  of  existing  literature  that  could  be 
relevant  to  understanding  the  intelligence  analysis  activities  and  the  personnel  attributes  required 
to  successfully  perform  these  activities.  The  study  team  conducted  a  three-part  review  that 
included  an  effort  aimed  at  understanding  the  intelligence  analysis  domain  and  required 
activities,  research  on  the  cognitive  attributes  that  lead  to  success  in  these  activities,  and  finally 
on  methods  for  measuring  the  attributes  and  quantifying  the  relationship  between  those  measures 
and  analyst  performance. 

As  noted  previously  in  this  report,  a  number  of  publications  attempt  to  describe  the  job  of  an  IA 
and  the  processes  undertaken  to  translate  raw  data  into  actionable  intelligence  information.  A 
universal  theme  across  the  literature  reviewed  for  this  effort  was  that  the  job  of  an  analyst  is 
varied,  complex,  and,  with  the  explosion  of  available  information,  requires  a  highly  skilled 
individual  to  make  sense  of  the  available  data.  One  of  the  key  pieces  of  literature  reviewed  in 
the  effort  to  understand  the  tasks  and  jobs  of  military  IAs  was  the  work  of  Seven  et  al.  (1991) 
that  developed  the  Job  Comparison  and  Analysis  Tool  (JCAT).  The  JCAT  was  developed  by  the 
Army  Research  Institute  (ARI)  to  assess  the  importance  of  various  abilities  to  the  activities 
performed  by  an  IA  (96  series  of  Military  Occupation  Specialty  -  Intelligence  MOSs).  The 
attributes  that  these  researchers  employed  to  identify  the  personnel  characteristics  associated 
with  “good”  IA  performance  were  derived  from  the  Manual  for  the  Ability  Requirements  Scales 
(MARS),  which  was  developed  by  Fleischman  (1991;  1992;  Fleischman  and  Quaintance,  1984). 
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The  work  to  develop  the  JCAT  provided  two  key  elements  for  the  current  effort.  First,  the 
personnel  attributes  identified  provided  a  validation  reference  for  the  characteristics  derived  from 
other  literature  sources.  Second,  additional  work  that  grew  out  of  JCAT  development  provided 
insight  into  the  specific  critical  tasks  for  an  all-source  analyst.  Specifically,  Knapp  and  Tillman 
(1998)  evaluated  IA  MOSs  using  JASS  (the  computerized  version  of  JCAT).  Their  task  analysis 
of  the  96B  MOS  determined  that  the  following  were  the  “critical  tasks”  for  that  MOS: 

•  Assess  incoming  information 

•  Determine  information  gaps 

•  Develop  a  doctrinal  template 

•  Develop  a  situation  template 

•  Develop  an  intelligence  briefing 

•  Develop  the  SITMAP 

•  Identify  high-payoff  targets 

•  Other 

•  Perform  interactive  input  processing 

•  Predict  potential  military  operations 

From  the  Knapp  and  Tillman  work,  in  conjunction  with  literature  descriptive  of  the  IA  tasks  and 
SME  input,  we  selected  the  following  tasks  as  being  representative  of  the  critical  tasks 
performed  by  35F  all-source  analysts: 

•  Analyze  map  data 

•  Determine  information  gaps  and  discrepancies 

•  ID  potential  targets 

•  Maintain  the  situation  map 

•  Evaluate  the  threat 

•  Perform  situation  development 

•  Define  the  operational  environment 

•  Describe  environmental  effects  on  the  threat  and  friendly  ops 

•  Conduct  fusion  of  information  from  multiple  intelligence  sources 
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2.1.1  Initial  Literature  Review 

The  second  element  of  the  literature  review  was  associated  with  identifying  the  cognitive 
attributes  that  are  descriptive  of  a  successful  analyst.  The  rationale  for  the  starting  point  for  this 
initial  review  was  based  on  our  conceptualization  of  potentially  useful  cognitive  theory  and 
research  focused  on  information  processing  (IP)  as  a  key  enabler  for  analysis,  reasoning,  and 
decision  making.  The  initial  list  of  cognitive  attributes  was  derived  from  our  preliminary  review 
of  previously  connected  efforts  that  were  similar  but  not  of  the  same  domain.  Of  specific  interest 
was  a  recent  ARI  research  report  describing  the  results  of  a  workshop  involving  the  science  of 
human  measures  (Goodwin  et  al.,  2009).  This  report  advocated  focusing  on  three  broad 
cognitive  attribute  areas  when  searching  for  ways  to  enhance  personnel  assessment,  training,  and 
professional  development:  What  Soldiers  “can  do,”  “want  to  do,”  and  “will  do.”  The  report  also 
advocated  looking  into  factors  and  processes  related  to  “mental  agility”  and  other  related 
problem-solving  capabilities  (see  Goodwin  et  al.,  2009,  pp  19-21).  This  preliminary  review  led 
the  study  team  to  conduct  an  initial  literature  review  that  looked  at  relevant  theory-driven 
cognitive/IP  research,  as  well  as  recent  advances  in  cognitive  neuroscience  research  related  to 
electroencephalogram  (EEG)  mapping.  The  most  notable  theoretical  approach  was  that  provided 
by  Robert  Sternberg’s  Triarchic  Theory  of  Intelligence  (1985;  1999),  which  included  promising 
but  limited  empirical  results  indicating  that  component  IP  activities  can  be  isolated  and  measured 
and  can  predict  performance  on  more  complex  cognitive  activities,  such  as  reasoning  and 
decision  making. 

The  results  of  this  literature  search  lead  to  other  potentially  useful  areas  of  cognitive  theory  and 
research,  including  neural  activity  (EEG)  mapping  and  Cognitive  Load  Theory  (CLT).  The 
reasons  for  reviewing  these  areas  were  threefold:  (1)  the  notion  of  metacognition,  or  knowledge 
about  one’s  cognitive  processes,  has  received  a  great  deal  of  attention  by  cognitive  theorists  and 
those  studying  strategic  thinking,  (2)  while  cognitive  attributes  have  traditionally  been  measured 
solely  through  one  of  a  wide  variety  of  intelligence  (cognitive)  ability  tests,  EEG  mapping 
offered  more  direct  evidence  of  cognitive  processing  activity  and  ability,  and  (3)  mental  effort  as 
a  potential  indicator  of  advanced  (cognitive)  strategy  use  is  one  area  of  research  thought  to  have 
relevance  for  this  project. 

A  brief  summary  of  the  findings  from  each  of  these  major  areas  of  the  literature  review  follows. 
A  more  extensive  discussion  of  the  initial  attribute-focused  literature  review  can  be  found  in 
appendix  B. 

Sternberg’s  Triarchic  Theory  of  Intelligence  (1985;  1999)  and  its  associated  Componential 
Analysis  approach  provide  an  interesting  and  potentially  useful  view  of  human  problem  solving 
(Laux  and  Lane,  1985).  Specifically,  intelligent  behavior  and  associated  problem-solving  ability 
is  viewed  in  terms  of  IP  components,  and  there  is  evidence  showing  IP  differences  are  related  to 
differences  in  performance  on  cognitive  tasks. 
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Studies  using  neural  activity  (EEG)  mapping  have  demonstrated  a  relationship  between  specific 
neurological  activity  and  short-term  (working)  memory  and  other  strategic  (cognitive) 
functioning  (Forstmann  et  al.,  201 1).  This  relationship  can  be  used  to  predict  performance  on 
complex  tasks. 

CLT  (Chandler  and  Sweller,  1991)  views  mental  “effort”  in  terms  of  demands  on  working 
memory  and  an  indicator  of  cognitive-processing  capability  and  efficiency.  One  approach  to 
measuring  CLT  involves  having  subjects  perform  two  tasks  simultaneously  (Paas  et  al.,  2003; 
Van  Merrienbohr  and  Sweller,  2005).  This  “dual  task”  methodology  is  problematic  because  of 
its  inability  to  definitively  determine  what  specific  cognitive  component(s)  are  being  measured. 
However,  the  study  team  recognized  that  there  was  a  possibility  of  combining  EEG  mapping 
with  CLT  methodologies  to  more  accurately  explore  the  notion  of  mental  effort  as  an  indicator  of 
IP  efficiency. 

By  combining  what  was  learned  from  related  studies  (e.g.,  Goodwin  et  al.,  2009;  Knapp  and 
Tillman,  1998;  Seven  et  al.,  1991)  with  insight  gleaned  from  the  initial  attribute-focused 
literature  search,  we  developed  a  list  of  potentially  important  cognitive  attributes  for  further 
study.  Table  1  presents  a  comparison  between  a  conceptual  view  based  on  work  presented  in  the 
ARI’s  Science  of  Human  Measures  Workshop  (Goodwin  et  al.,  2009),  the  JCAT-related  work, 
and  our  conceptualization  framed  by  the  initial  literature  review  efforts. 

Table  1.  Cognitive  attribute  comparison  approach. 


Cognitive  / 

Initial 

Selected 

Information*  Processing 
Function 

Attributes 

JCAT  Attributes 

Core  IP  Function 

S-T  Memory 

Memorization 

(Identify/Store  Information) 

L-T  Memory 

Attention  Management 

Selective  Attention 

Pattern  Recognition 

Time  Sharing 

Mediating  IP  Function 

(Manipulate  Information) 

Situational  Awareness 

Deductive  Reasoning 

Problem  Solving 

Inductive  Reasoning 

Decision  Making 

Problem  Sensitivity 

Adaptability 

Flexibility  of  Closure 

Creativity 

Fluency  of  Ideas 

Executive  IP  Function 

Automaticity 

Originality 

(Strategy  Development  &  Use  - 

Meta-Cognition 

TBD 

Involves  Planning,  Executing, 
Monitoring) 

Perseverance/Resilience 
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The  cognitive  abilities  (attributes)  required  to  perform  the  35F  IA  job  were  identified  based  on 
the  literature  and  SME  input  (as  described  in  section  2.3).  The  attributes  determined  as  required 
to  perform  critical  35F  tasks  were  as  follows: 

1.  Memorization 

2.  Selective  attention 

3.  Time-sharing 

4.  Deductive  reasoning 

5.  Inductive  reasoning 

6.  Problem  sensitivity 

7.  Flexibility  of  closure 

8.  Fluency  of  ideas 

9.  Originality 

SMEs  subsequently  identified  perseverance  as  a  tenth  attribute  of  importance  to  add  to  this  list. 

The  final  element  of  the  literature  review  was  focused  on  identifying  studies  where  these  critical 
attributes  had  been  assessed  and  where  they  were  used  to  predict  performance  on  tasks  that  were 
related  to  the  critical  tasks  just  listed. 

2.1.2  Attribute  Literature  Review 

The  final  aspect  of  the  literature  review  conducted  for  this  effort  was  a  formal  two-part  literature 
search  guided  by  the  attributes  derived  from  the  literature  and  later  validated  by  SME  consensus, 
as  important  for  IA  performance.  This  effort  represents  the  primary  focus  of  the  overall  project 
and  was  aimed  at  understanding  both  how  personnel  attributes  can  be  measured  and  how  the 
attributes  can  be  quantitatively  linked  to  performance  on  IA  activities.  To  establish  this 
quantitative  link  between  attribute  levels  and  analyst  task  performance,  a  meta-analytic  approach 
was  used  targeting  correlational  studies  examining  the  relationship  between  scores  on 
standardized  tests  measuring  the  attributes  and  performance  on  experimental  tasks  or  tests  that 
share  characteristics  similar  to  intelligence  analysis  tasks. 

We  began  by  reviewing  the  Mental  Measurements  Yearbook  (Spies  et  al.,  2010)  for  standardized 
tests  related  to  the  personnel  attributes  of  interest.  Pertinent  synonyms  for  each  attribute  of 
interest  were  generated  for  keyword  searches.  For  example,  to  identify  studies  measuring  the 
attribute  “originality,”  search  terms  included  “originality”  and  “creativity,”  and  for  “fluency  of 
ideas,”  search  terms  included  “fluency  of  ideas”  and  “ideational  fluency.”  The  list  of  attributes 
and  alternate  key  words  was  used  to  identify  measures  in  the  Mental  Measurements  Yearbook 
where  any  of  the  keywords  appear  in  the  test  name,  test  purpose,  or  test  index.  This  search 
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generated  a  list  of  available  standardized  tests  that  measure  the  cognitive  attributes  required  for 
successful  IA  performance.  A  full  list  of  the  standardized  tests  by  attribute  is  provided  in 
appendix  C. 

The  list  of  standardized  tests  used  to  either  partially  or  wholly  measure  the  attributes  that  had 
been  identified  as  important  to  IA  performance  allowed  us  to  target  the  next  round  of  literature 
searches  on  empirical  studies  referencing  the  specific  standardized  tests,  which  were  likely  to 
have  measured  the  personnel  attribute  as  a  covariate  to  some  performance  measure  of  interest. 
Only  studies  that  tested  normal  adult  populations  resembling  IA  personnel  were  examined 
further.  Search  results  were  excluded  if  the  population  was  specific  to  young  children  or  older 
adults,  or  if  the  primary  purpose  was  diagnosis  of  deficiencies  in  the  attribute. 

We  utilized  the  database  services  of  EBSCO  Publishing  to  search  seven  relevant  databases: 
Academic  Search  Premier,  Business  Source  Premier,  CINAHL,  ERIC,  PsychARTICLES, 
PsychINFO,  and  Social  Sciences  Abstracts.  All  standardized  tests  identified  for  each  attribute 
were  individually  searched  in  “all  text,”  and  the  results  were  saved  by  test  name.  Results  were 
then  reviewed  for  relevance  and  to  ensure  they  met  the  following  set  of  requirements: 

•  The  study  must  report  the  test  used  for  measurement  of  attributes. 

•  The  study  must  report  the  experimental  task,  test,  or  activity  also  performed  by  the 
participants. 

•  The  study  must  report  a  measure  of  the  relationship  (preferably  Pearson  product  moment) 
between  measured  quantity  of  an  attribute  and  measured  performance  on  the  experimental 
task. 

By  filtering  the  search  results  based  on  these  criteria,  we  identified  a  subset  of  the  available 
literature  for  the  meta-analytic  review.  This  subset  of  literature  was  then  mined  for  quantitative 
performance  relationships.  An  Excel  dataset  was  created  to  record  the  following  information  for 
each  included  study: 

•  The  full  citation 

•  Attribute  measured  (i.e.,  deductive  reasoning) 

•  The  measurement  instrument  (i.e.,  Cornell  Critical  Thinking  Test) 

•  A  description  of  test  correlate  (i.e.,  experimental  problem-solving  task) 

•  The  type  of  effect  measure  (i.e.,  d’,  t,  r,  etc.) 

•  A  conversion  of  the  test  static  to  r 

•  The  group  sample  size  of  the  study 
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The  contents  of  the  MS  Excel  spreadsheet  provided  the  basis  for  establishing  effects  relevant  to 
predicting  performance  on  IA  tasks  from  performance  on  standardized  attribute  measures. 

However,  translating  the  data  in  this  raw  form  into  something  that  could  be  incorporated  into  our 
model  (and  modeling  environment)  was  complex.  Our  modeling  environments  use  a  task 
demand  approach  to  build  subtask  networks.  The  performance  relationships  from  the  literature 
were  in  raw  experimental  task  form,  not  in  task  demand  composition  format.  Consequently, 
each  test  correlate  that  standardized  test  scores  had  been  regressed  upon  in  the  studies  was 
characterized  in  terms  of  the  task  demand  taxonomy  as  might  be  done  by  a  modeler  attempting  to 
model  the  experimental  tasks.  A  10-task  demand  taxonomy  based  on  the  10  IA  attributes  was 
used  to  characterize  experimental  tasks.  This  step  required  the  team  to  make  subjective  but 
informed  judgments  about  the  task  demands  that  best  described  the  experimental  tasks.  It 
ultimately  allowed  us  to  translate  the  raw  effects  presented  in  the  literature  into  an  initial  set  of 
data  comprising  sets  of  associated  concepts  that  already  existed  in  our  modeling  environments, 
i.e.,  personnel  attribute  or  standardized  test  scores  and  task  “taxons”  or  the  experimental  tasks 
characterized  by  taxons.  The  resulting  data  set  consisted  of  correlations  between  attributes  and 
typologies  of  task  demands. 

2.1.3  Task  Typology  and  Attribute  Relationship  to  Task  Taxon  Demands 

Once  the  task  demand  mapping  was  complete,  we  were  confronted  with  the  issue  of  whether  or 
how  to  reduce  the  identified  effects/relationships  into  a  one-to-one  attribute  to  taxon  relationship. 
For  example,  say  hypothetically  that  every  experimental  problem-solving  task  was  characterized 
with  deductive  reasoning  (DR),  organization  of  information  (01),  and  memory  (M)  taxons.  This 
results  in  problem-solving  tasks  being  characterized  by  a  DR-OI-M  typology  of  taxons. 
Furthermore,  if  we  identified  six  studies  in  the  literature  that  correlated  a  standardized  test  of 
flexibility  of  closure  (FC)  with  “problem-solving”  task  performance,  then  we  could  compute  the 
mean  correlation  between  FC  and  the  DR-OI-M  typology.  If  this  performance  effect  was 
embedded  in  the  modeling  environment,  then  whenever  a  task  was  modeled  and  characterized  by 
DR,  01,  and  M  taxons,  we  could  assume  to  know  the  approximate  nature  of  the  relation  between 
an  individual’s  FC  ability  and  that  specific  modeled  task’s  performance. 

Typically,  discrete  event  simulation  modeling  tools  such  as  Micro  Saint  Sharp  and  IMPRINT 
want  the  personnel  attribute-to-task  relationship  to  be  represented  at  a  more  granular  level  than  a 
taxon  typology;  and  consequently,  there  is  the  desire  to  further  reduce  the  attribute-to-task 
relationship  to  the  individual  taxon  level.  Because  we  are  starting  with  mean  correlation 
coefficients,  these  reductions  to  individual  taxons  are  confronted  by  issues  of  (multiple) 
colinearity  because  we  do  not  know  whether  the  taxons  that  characterize  a  task  are  all 
contributing  to  task  performance,  and  if  so,  how  much  each  contributes,  and  whether  their 
individual  contributions  are  intercorrelated.  For  example,  if  both  inductive  reasoning  and 


10 


deductive  reasoning  ability  predict  performance  on  the  Evaluate  the  Threat  task,  we  must  ask 
whether  their  contribution  is  independent  (additive)  or  does  either  of  them  predict  performance 
on  Evaluate  the  Threat  performance  as  well  as  including  both  of  them  as  predictors?  The  issue 
of  multicollinearity  exists  both  for  the  task  typologies  (when  a  whole  task  is  modeled  at  the 
subtask  level  with  a  typology  and  then  recombined)  and  when  tasks  are  described  by  multiple 
taxons  (whenever  tasks  and  subtasks  are  modeled  with  a  combination  of  taxons).  Because  of 
these  issues,  we  chose  not  to  make  the  assumptions  necessary  for  the  singular  taxon  reduction 
and  instead  chose  to  leave  the  tasks  at  the  typology  level  of  task  demand  characterization. 

Following  the  general  meta-analytic  approach  for  meta-analysis  of  correlation  measures  in 
Borenstein  et  al.  (2009),  the  dataset  was  analyzed  using  Comprehensive  Meta  Analysis  software 
(http://www.meta-analysis.com/).  Here,  the  correlation  coefficient  (r)  and  number  of  study 
participants  (n)  were  transformed  to  Fisher’s  Z,  and  summary  effects  and  standard  error  (SE) 
were  computed  for  each  demand  typology  within  individual  personnel  attributes.  The  results 
were  compiled  into  an  attribute  by  task  demand  typology  matrix  (see  appendix  D). 

2.2  Key  Cognitive  Attributes/Abilities 

Abilities  and  aptitudes  are  cognitive  or  mental  characteristics  or  attributes  that  affect  the 
potential  to  learn  or  to  perform.  Cognitive  abilities  can  be  thought  of  in  broad  terms  (e.g., 
intelligence)  or  in  terms  of  specialized  abilities,  such  as  verbal,  spatial,  memory,  reasoning, 
problem  solving,  and  psychomotor  ability.  Aptitudes  may  be  more  broadly  conceptualized  than 
abilities  and  may  include  any  number  of  individual-differences  factors,  such  as  motivation,  that 
influence  the  likelihood  of  performing  successfully.  In  1984,  Fleischman  described  tasks  in 
terms  of  performance  taxonomies  that  include  both  abilities  and  aptitudes  (Fleischman  and 
Quaintance,  1984). 

ARI  developed  JCAT  in  order  to  identify  MOS  capabilities  and  Intelligence  Electronic  Warfare 
(IEW)  system  demands  “in  terms  of  abilities,  skills,  and  intelligence  production  activities” 

(Seven  et  al.,  1991).  The  origin  of  the  list  of  ability  requirements  from  which  the  abilities  in  the 
JCAT  were  derived  was  MARS,  which  was  developed  by  Fleischman  (1991;  1992;  Fleischman 
and  Quaintance,  1984)  and  ARI.  The  MARS  list  of  abilities  has  been  widely  used  and  validated 
(Seven  et  al.,  1991).  The  ARI  research  included  50  abilities  from  the  Fleischman  list  of  abilities 
that  they  considered  to  be  potentially  important  to  IAs  (see  table  2  for  the  list  of  abilities 
included  in  the  JCAT). 

In  order  to  reduce  the  list  for  the  current  study  to  those  attributes  that  were  deemed  important  to 
all-source  IAs,  we  selected  only  those  abilities  that  were  selected  by  eight  or  nine  (of  nine)  IAs  in 
the  Seven  et  al.  (1991)  study  as  being  an  important  attribute  of  the  96B  (now  35F)  IA  job,  or 
which  were  rated  by  the  nine  96B  experts  as  greater  than  four  (of  seven)  in  importance  to  the  IA 
task.  This  resulted  in  a  list  of  20  attributes  that  we  considered  relevant  to  performance  of  the 
all-source  35F  IA  position  and  which  could  be  assessed  using  a  cognitive  ability  test.  The 
ratings  from  the  Seven  et  al.  (1991)  study  are  shown  in  table  2  where  aptitudes  in  red  were  not 
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Table  2.  Aptitudes  from  the  JCAT  that  were  assessed  in  the  Seven  et  al.  (1991)  study. 


Aptitude/Ability 
for  96B 
(35F) 

Named  by  Eight  or 
Nine  SMEs  or  Rated 
Importance 

4  or  >4 

(red  =  <8  SMEs) 

Rated  Importance  to 
Job  of  7 
(Mean/SD) 

Oral  comprehension 

9 

5. 9/1.0 

Written  comprehension 

9 

6. 4/0. 8 

Oral  (speech)  expression 

9 

6. 2/0. 8 

Written  expression 

9 

6. 0/0. 9 

Memorization 

9 

5. 6/1. 5 

Problem  sensitivity 

9 

5. 1/1.6 

Originality 

8 

4.8/1. 3 

Fluency  of  ideas 

9 

4. 8/2.2 

Flexibility  of  closure 

8 

4.9/2. 1 

Selective  attention 

8 

5. 3/1.5 

Spatial  orientation 

9 

5.2/1 .4 

Visualization 

9 

5. 8/1.4 

Inductive  reasoning 

8 

5. 3/1.7 

Category  flexibility 

6 

5.2/1. 8 

Deductive  reasoning 

9 

5.9/1. 3 

Information  ordering 

8 

5. 0/1. 7 

Math  reasoning 

7 

4. 5/1. 7 

Number  facility 

8 

3. 9/1. 7 

Time  sharing 

8 

5. 8/1.2 

Speed  of  closure 

8 

5.5/2. 1 

Perceptual  speed  and  accuracy 

7 

4.4/1. 8 

Near  vision 

6 

4. 3/1. 9 

Far  vision 

6 

4.1/1. 7 

Visual  color  discrimination 

7 

4. 2/2.0 

General  hearing 

7 

4. 1/1.7 

Auditory  attention 

6 

5. 0/0. 6 

Finger  dexterity 

8 

4.7/1. 4 

Manual  dexterity 

6 

4.8/1. 1 

Arm-hand  steadiness 

6 

4.5/11.8 

Multi-limb  coordination 

5 

4. 1/1.9 

Extent  flexibility 

6 

4.1/1. 8 

Cross  body  coordination 

6 

4.2/1. 9 

Static  strength 

7 

4.0/1 .4 

Trunk  strength 

4 

4. 0/2. 2 

Stamina 

6 

4. 3/2. 5 

rated  as  important  by  eight  or  nine  analysts  in  the  expert  group,  but  importance  was  rated  >4  by 
the  remaining  experts.  This  list  was  mapped  against  our  initial  conceptual  attributes  list  based  on 
other  intelligence-related  efforts  and  our  review  of  the  literature  surrounding  theories  of  human 
intelligence  (e.g.,  Sternberg’s  theories),  innovations  in  EEG  measurement  of  cognitive  attributes, 
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and  cognitive  load  theory  as  previously  described.  Because  of  the  high  degree  of  conceptual 
overlap  between  our  initial  list  and  that  generated  from  the  work  of  Seven  et  al.,  we  elected  to 
use  the  attributes  from  the  JCAT  work  because  of  the  rigorous  psychometric  validation  effort 
that  was  undertaken  on  the  JCAT. 

Based  on  these  analyses  and  the  criterion  that  the  aptitude  should  be  measurable  by  a  cognitive 
test,  we  selected  the  following  20  aptitudes  as  those  that  we  believed  could  be  assessed  for  use  as 
predictors  of  IA  performance. 

Definitions  of  cognitive  attributes  from  MARS: 

1.  Oral  Comprehension:  The  ability  to  listen  to  and  understand  words  and  sentences. 

2.  Written  Comprehension:  The  ability  to  understand  written  words,  sentences,  and  paragraphs. 

3.  Oral  Expression:  The  ability  to  use  words  or  sentences  in  speaking  so  that  others  will 
understand. 

4.  Written  Expression:  The  ability  to  use  words  or  sentences  in  writing  so  that  others  will 
understand. 

5.  Memorization:  An  ability  to  remember  information  such  as  words,  numbers,  pictures,  and 
procedures. 

6.  Problem  Sensitivity:  The  ability  to  tell  when  something  is  wrong  or  is  likely  to  go  wrong. 

It  does  not  involve  solving  the  problem,  only  recognizing  there  is  a  problem. 

7.  Originality:  The  ability  to  produce  unusual  or  clever  ideas  about  a  given  topic  or  situation. 
It  is  the  ability  to  invent  creative  solutions  to  problems  or  develop  new  procedures  for 
situations  in  which  standard  procedures  do  not  apply  or  are  not  working. 

8.  Fluency  of  Ideas:  The  ability  to  come  up  with  a  number  of  ideas  about  a  topic  (the  number 
of  ideas  is  important,  not  their  quality,  correctness,  or  creativity). 

9.  Flexibility  of  Closure:  That  ability  to  identify  or  detect  a  known  pattern  (a  figure,  object, 
word,  or  sound)  that  is  hidden  in  other  distracting  material. 

10.  Selective  Attention:  The  ability  to  concentrate  on  a  task  over  a  period  of  time  without 
being  distracted. 

11.  Spatial  Orientation:  The  ability  to  tell  where  you  are  in  relation  to  the  location  of  some 
object  or  to  tell  where  the  object  is  in  relation  to  you. 

12.  Visualization:  The  ability  to  imagine  how  something  will  look  when  it  is  moved  around  or 
when  its  parts  are  moved  or  rearranged.  It  requires  the  forming  of  mental  images  of  how 
patterns  or  objects  would  look  after  certain  changes,  such  as  unfolding  or  rotation.  One  has 
to  predict  how  an  object,  set  of  objects,  or  pattern  will  appear  after  the  changes  are  carried 
out. 
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13.  Inductive  Reasoning:  The  ability  to  combine  pieces  of  information  to  form  general  rules  or 
conclusions  (includes  finding  a  relationship  among  seemingly  unrelated  events). 

14.  Deductive  Reasoning:  The  ability  to  apply  general  rules  to  specific  problems  to  produce 
answers  that  make  sense. 

15.  Category  Flexibility:  The  ability  to  produce  many  rules  so  that  each  rule  tells  how  to  group 
a  set  of  things  in  a  different  way.  Each  different  group  must  contain  at  least  two  things 
from  the  original  set  of  things. 

16.  Information  Ordering:  The  ability  to  follow  correctly  a  rule  or  set  of  rules  to  arrange  things 
or  actions  in  a  certain  order.  The  rule  or  set  of  rules  must  be  given.  The  things  or  actions 
to  be  put  in  order  can  include  numbers,  letters,  words,  pictures,  procedures,  sentences,  and 
mathematical  or  logical  operations. 

17.  Mathematical  Reasoning:  The  ability  to  understand  and  organize  a  problem  and  then  select 
a  mathematical  method  or  formula  to  solve  the  problem.  It  encompasses  reasoning  through 
mathematical  problems  to  determine  appropriate  operations  that  can  be  performed  to  solve 
problems.  It  also  includes  the  understanding  or  structuring  of  mathematical  problems.  The 
actual  manipulation  of  numbers  is  not  included  in  this  ability. 

18.  Number  Facility:  Involves  the  degree  to  which  adding,  subtracting,  multiplying,  and 
dividing  can  be  done  quickly  and  correctly.  These  can  be  steps  in  other  operations  like 
finding  percentages  and  taking  square  roots. 

19.  Time  Sharing:  The  ability  to  shift  back  and  forth  between  two  or  more  activities  or  sources 
of  information  (such  as  speech,  sounds,  touch,  or  other  sources). 

20.  Speed  of  Closure:  Involves  the  degree  to  which  different  pieces  of  information  can  be 
combined  and  organized  into  one  meaningful  pattern  quickly.  It  is  not  known  beforehand 
what  the  pattern  will  be.  The  material  may  be  visual  or  auditory. 

From  this  list,  the  following  nine  attributes  were  selected,  based  on  SME  input,  as  being  critical 


for  IA  performance: 

•  Memorization 

•  Selective  Attention 

•  Time- sharing 

•  Deductive  Reasoning 

•  Inductive  Reasoning 

•  Problem  Sensitivity 

•  Flexibility  of  Closure 

•  Fluency  of  Ideas 

•  Originality 
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These  nine  key  attributes  were  used  to  describe  both  the  tasks  a  35F  all-source  analyst  performs 
in  support  of  mission  objectives  and  the  individual  who  performs  the  tasks.  That  is,  each  task  the 
analyst  performs  requires  one  or  more  of  the  nine  attributes,  and  the  degree  to  which  the 
individual  performing  that  task  possesses  a  sufficient  quantity  of  the  required  attribute(s)  impacts 
his  or  her  ability  to  perform  the  task. 

2.3  Tasks  of  an  All-Source  Intelligence  Analyst 

In  order  to  quantify  the  relationship  between  attribute  levels  and  task  performance,  we  needed  to 
understand  the  intelligence  analysis  process,  the  tasks  that  compose  this  process,  and  the 
characteristics  of  the  demands  of  the  tasks  in  terms  of  analyst  aptitude.  To  develop  this 
understanding,  we  reviewed  Army  doctrine,  manuals,  and  training  related  publications  to 
validate  the  critical  tasks  of  the  35F  as  initially  identified  through  literature  review. 

Field  Manual  34-3,  Intelligence  Analysis,  states  that  an  IA  converts  combat  information  into 
intelligence  by  following  a  four-step  process:  observation,  assessment,  analysis,  and  synthesis 
(Headquarters,  1990).  Field  Manual  2-0,  Intelligence,  defines  analysis  as  “the  process  by  which 
collected  information  is  evaluated  an  integrated  with  existing  information  to  product  intelligence 
that  describes  the  current  -  and  attempts  to  predict  the  future  -  impact  of  the  threat,  terrain  and 
weather,  and  civil  considerations  on  operations”  (Headquarters,  2010a).  According  to  Land 
(2004),  four  analytical  tasks  make  up  the  analysis  and  synthesis  process: 

1.  Identify  the  problem 

2.  Conduct  background  research 

3.  Identify  intelligence  production  requirements  and  options 

4.  Request  information 

As  Land  (2004)  notes,  FM  34-3  describes  a  set  of  dynamic  and  interactive  analytical  skills 
needed  by  IAs: 

1.  Understanding  the  analytic  objective — understanding  the  desired  end-state  of  reaching  a 
predictive  conclusion  about  the  threat. 

2.  Establishing  the  baseline — the  analyst  collects  and  organizes  the  pertinent  facts  about  a 
given  situation  or  problem. 

3.  Formulating  the  hypothesis — the  analyst  develops  hypotheses  to  account  for  any 
information  gaps  and  questions  resulting  from  a  review  of  the  baseline  information 
regarding  the  environment,  its  effects,  and  the  threat.  The  analyst  ensures  that  his 
hypotheses  address  the  correct  problem. 

4.  Testing  the  hypothesis — the  analyst  will  either  confirm  or  deny  his  hypothesis  by 
developing  observable  indicators  that  can  be  collected. 
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5.  Recognizing  uncertainties — an  IA’s  primary  mission  is  to  decrease  uncertainty  by 
confirming  or  denying  hypotheses  about  the  environment  or  threat. 

The  35F  MOS  designates  the  all-source  IA.  FM  2-0  (2010a)  defines  all-source  intelligence  as 
“[t]he  intelligence  discipline  concerned  with  all-source  products  and  the  processes  used  to 
produce  them.”  “Analysis  is  achieved  through  the  reduction  of  information  to  its  basic 
components.  Each  of  these  components  is  then  examined  to  determine  its  nature,  proportion, 
function,  and  interrelationships.  .  .  .  The  analyst  examines,  assesses,  and  compares  bits  and 
pieces  of  raw  information,  and  then  synthesizes  those  findings  into  an  intelligence  product 
reflecting  an  adversary’s  capabilities  and  vulnerabilities.  Most  intelligence  analysis  is  predictive 
in  nature  .  .  .”  (FM  34-130  Draft  2000,  2-1  as  cited  in  Land,  2004). 

In  order  to  accomplish  the  high-level  intelligence  analysis  goals  described  by  these  field 
manuals,  the  35F  MOS  IA  performs  a  set  of  discrete  tasks.  These  tasks  were  identified  in 
training  course-related  materials  provided  by  the  Intelligence  Center  of  Excellence  in  support 
of  this  project  effort  and  from  relevant  literature.  A  Personnel  Supervision  and  Management 
(PS&M)  list  of  the  35F  Skill  Level  Tasks  and  a  Skills  Crosswalk 

(35FCrosswalkCTLAnalysisDIF29JAN10a.xls)  was  provided  to  the  research  team.  From  these 
materials,  the  set  of  specific  tasks  performed  by  the  analyst  was  identified. 

In  order  to  define  the  research  effort  for  this  project  and  focus  the  modeling  effort,  the  team 
needed  to  identify  a  prioritized  list  of  analyst  tasks  to  be  the  subject  of  a  further  detailed  analysis. 
In  order  to  accomplish  this  we  leveraged  the  difficulty,  importance,  and  frequency  (DIF)  ratings 
provided  as  part  of  the  Skills  Crosswalk  document.  By  averaging  the  ratings,  we  were  able  to 
develop  a  priority  metric  for  the  tasks.  The  list  of  tasks  was  then  compared  to  the  tasks  listed  in 
other  documents  to  identify  common  elements  between  the  task  lists. 

One  highly  rated  task  that  appeared  in  all  documents  was  the  Evaluate  the  Threat  activity.  Given 
the  commonality  of  this  task  across  all  provided  materials,  its  importance  based  on  DIF  ratings, 
and  the  obvious  analytic  nature  of  the  task,  we  elected  to  use  it  as  the  basis  for  our  modeling  and 
continued  research  efforts.  The  detailed  subtasks  that  make  up  this  task  and  the  data  collected 
from  SMEs  on  task  performance  were  captured  in  the  model  and  are  described  later  in  this 
report. 

In  order  to  identify  the  specific  attribute  requirements  for  successful  performance  as  an  analyst, 
the  team  needed  to  map  the  specific  attributes  required  for  task  performance  to  the  tasks  that 
analysts  perform.  A  number  of  researchers  have  analyzed  the  KSA  requirements  associated  with 
the  IA  job.  Knapp  and  her  colleagues  (Knapp  and  Tillman,  1988;  Seven  et  al.,  1991)  described 
the  IA  functions  and  tasks  using  Fleishman’s  task  taxonomy  and  developed  a  tool  for 
determining  which  KSAs  are  associated  with  specific  IA  jobs.  The  resulting  tool,  the  JCAT,  was 
developed  to  identify  MOS  capabilities  and  IEW  system  demands  in  terms  of  abilities,  skills,  and 
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intelligence  production  activities.  The  JCAT  was  subsequently  used  to  collect  abilities  and  skills 
data  for  the  seven  MOSs  comprising  what  is  now  the  35  MOS  group  of  IAs. 

In  an  effort  to  validate  the  tasks  derived  from  the  PS&M  documentation,  we  revisited  a  number 
of  publications,  including  FM  2-0  Intelligence  (2010)  and  FM  34-3  Intelligence  Analysis  (1990), 
which  describe  the  intelligence  process  performed  by  the  IA  in  detail.  Additionally,  Krizan 
(1999)  defined  the  KSAs  that  describe  the  IA  job  while  Heuer  (1999)  and  others  have  addressed 
the  psychological  issues  associated  with  IA  performance,  such  as  perception,  memory,  and 
cognitive  biases.  Other  authors  have  described  the  task  of  IA  as  a  means  of  assisting  analysts  to 
perform  their  jobs  more  effectively  (e.g.,  Clawson,  2008;  Connors  et  al.,  2004;  Land,  2004).  We 
compared  these  publications  with  tasks  derived  from  the  training-related  materials  provided  by 
the  Intelligence  Center  of  Excellence  and  information  contained  in  the  JCAT.  The  results  of  this 
comparison  are  shown  in  table  3.  There  was  considerable  agreement  among  these  experts  and 
the  other  authors  just  cited  concerning  what  comprises  the  IA  job.  Based  on  these  analyses,  and 
those  previously  described,  we  accepted  the  12  tasks  in  the  job  description  from  the  35F 
crosswalk  duty  table  as  representative  of  the  critical  functions/tasks  performed  by  the  35F. 

Table  3.  35F  task  crosswalk. 


301 -35I-xxx-.sk 

Modified  Duty 

35F  Crosswalk 

Duties/JCAT 

Duties  Intelligence 
Essentials  for 
Everyone  (Krizan) 

Define  the  operational  environment 

IPB;  Planning  (353F40 
only) 

Planning;  set  up, 

maintenance, 

preparing 

Describe  environmental  effects  on 
threat  and  friendly  operations 

Analysis;  Advance  map 
reading;  IPB 

Analyze/exploit  data 

Data  monitoring; 

Data  collection 

Data  analysis 

Evaluate  the  threat 

ISR;  Analysis 

Collect  data; 
Analyze/exploit  data 

Data  monitoring; 

Data  collection 

Data  analysis 

Determine  threat  courses  of  action 

ISR;  Analysis; 

Targeting 

Interpret  data 

Data  analysis 

Perform  situational  development 

ISR;  Analysis;  IPB 

Interpret  data 

Data  analysis 

Present  intelligence  findings 

Reports  and 
dissemination 

Disseminate  data 

Data  interpretation/ 
communication 

Draft  ISR  synchronization  products 

ISR;  Analysis 

Prepare  outputs 

Data  analysis 

Develop  ISR  synchronization 
products 

ISR;  Analysis 

Prepare  outputs 

Computer  utilization 

Perform  ISR  synchronization 

ISR;  Analysis 

Prepare  outputs 

Computer  utilization 

Integrate  ISR  synchronization 

ISR;  Analysis 

Prepare  outputs 

Computer  utilization; 
coordination 

Provide  intelligence  support  to 
targeting 

Reports  and 
dissemination 

Disseminate  data 

Data  interpretation/ 

communication; 

coordination 

Provide  intelligence  support  to 
training 

Provide  intelligence  to 
combat  assessments; 
reports  and  dissemination 

Disseminate  data 

Coordination 
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2.4  Subject  Matter  Expert  Data  Collection 

Having  identified  the  attributes  we  believed  to  be  characteristic  of  a  successful  IA  from  the 
literature  review  effort  and  selected  a  target  analyst  task  for  analysis,  we  then  interacted  with  IA 
SMEs  to  confirm  our  initial  findings  and  collect  the  data  needed  to  move  on  to  the  next  phase  of 
the  project.  Two  members  of  the  study  team  held  interviews  with  intelligence  analysis  SMEs  at 
the  U.S.  Army  Intelligence  Center  at  Fort  Huachuca,  AZ,  on  12-14  July  2011.  The  primary 
objectives  were  to  (1)  identify  and  then  further  decompose  one  or  more  35F  (all-source  IA)  tasks 
in  order  to  focus  the  modeling  effort,  (2)  confirm  the  relevance  of  the  nine  cognitive  attributes 
previously  identified  as  critical  for  performing  their  associated  job-related  tasks,  and  (3)  use  the 
decomposed  35F  task(s)  to  determine  the  degree  to  which  these  attributes  get  employed  by  an 
analyst  when  completing  the  task(s). 

Over  the  course  of  the  interviews,  the  study  team  interacted  with  four  senior  IA  SMEs.  The 
overall  plan  of  action  by  the  study  team  was  to  revalidate  that  the  nine  cognitive  attributes  were 
still  relevant  for  completing  35F  job  tasks  and  to  identify  other  relevant  attributes  the  SMEs 
considered  to  be  critically  important  for  successful  performance.  The  study  team  worked  with 
the  SMEs  to  identify  one  or  two  tasks  considered  to  be  of  high  interest  and  representative  in 
terms  of  their  inherent  utilization  of  cognition-relevant  skills,  such  as  analysis,  reasoning,  and 
decision  making. 

Once  one  or  two  tasks  of  primary  interest  were  identified,  the  study  team  worked  with  the  SMEs 
to  decompose  the  primary  interest  task(s)  in  order  to  determine  important  task  characteristics. 
These  characteristics  included  (1)  what  constituted  successful,  as  well  as  subpar,  performance  for 
a  given  task  or  task  element;  (2)  how  long  did  a  task  normally  take  to  complete;  and  (3)  what 
interdependencies  existed  between  tasks.  To  the  extent  possible,  example  situations  were 
elicited  from  the  SMEs  as  a  means  for  both  the  SMEs  and  the  study  team  members  to  explore  the 
task  characteristics  in  further  detail.  These  characteristics  were  important  for  developing  a  useful 
exploratory  model  for  this  project.  The  manner  in  which  these  tasks  were  constructed  into  the 
human  performance  model  developed  for  this  effort  is  described  in  section  2.5,  and  the  detailed 
task  information  can  be  found  in  appendix  E  of  this  report.  At  no  time  during  the  interview 
process  did  the  level  of  discussion  include  sensitive  or  classified  information. 

The  final  step  in  the  SME  interview  process  was  to  determine  what  selected  cognitive  attributes 
were  needed  to  successfully  perform  a  given  task  and/or  task  element.  To  do  this,  a  seven-point 
Likert-like  scale  was  used  in  conjunction  with  a  task-by-attribute  matrix  that  was  developed 
based  on  the  results  of  the  SME  interview  process.  The  results  are  shown  in  table  E-l.  Table 
E-2  contains  the  duration  of  subtasks  and  performance  criteria  for  the  steps  of  the  Evaluate  the 
Threat  task. 

During  interaction  with  the  SMEs  at  Fort  Huachuca,  an  additional  (tenth)  attribute,  along  with  a 
description,  was  identified  as  critical  by  the  SMEs: 
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Perseverance:  An  ability  to  want  to  coax  the  best  possible  answer  out  of  a  situation  and  act 
on  it  in  an  appropriately  timely  manner  knowing  the  situation  and  associated  repercussions. 

The  need  for  perseverance  was  a  consistent  theme  across  the  SMEs  interviewed  for  this  effort, 
and  as  such,  we  included  the  attribute  in  the  subsequent  analyses  for  the  project.  However,  it  is 
interesting  to  note  that  this  particular  attribute  was  not  highlighted  in  the  literature  review 
conducted  to  develop  the  attribute  list.  Multiple  potential  reasons  for  the  disconnection  between 
the  literature  and  SME  opinions  are  plausible.  First,  much  of  the  literature  we  reviewed  was 
focused  on  the  initial  analysis  activities,  that  is,  the  literature  sought  to  describe  what  was 
required  to  collect,  filter,  and  transform  data  into  information  to  create  a  product.  Because  the 
literature  focused  on  the  initial  analysis  efforts,  the  resilience  and  persistence  that  characterizes 
perseverance  would  likely  not  come  into  play  in  the  initial  analytic  efforts.  Second,  because 
much  of  the  literature  is  concerned  with  intelligence  product  generation  as  a  measure  of 
performance,  it  is  reasonable  to  believe  that  perseverance  is  not  as  important  in  creating  initial 
products  but  would  increase  in  importance  with  a  need  to  revise  or  update  products  to  achieve  a 
greater  level  of  understanding  of  the  situation  being  analyzed.  Regardless,  it  is  clear  that  expert 
analysts  believe  that  to  be  effective  at  the  job  of  an  MOS  35F,  a  sort  of  resilience  or  persistence 
of  effort  is  required,  and  this  attribute  was  included  in  our  task  rating  and  modeling  activities. 

In  addition  to  perseverance,  through  the  discussions  with  the  SMEs,  two  additional 
characteristics  were  identified  as  being  consistent  with  an  IA  who  possesses  desirable 
performance  traits.  The  SMEs  consistently  cited  a  need  for  the  ability  to  organize  information 
and  to  be  an  effective  collaborator.  Descriptions  of  these  were  formed  and  are  as  follows: 

•  Organization  of  Information:  An  ability  to  come  up  with  a  system  or  method  to  save  and 
store  information  for  retrieval  and  communication. 

•  Collaboration:  An  ability  and  willingness  to  share  and  debate/discuss  ideas  with  others. 

Based  on  SME  input,  it  is  apparent  that  these  abilities  are  vital  elements  for  success  as  a  35F 
all-source  analyst.  However,  for  several  reasons,  within  the  current  effort  we  elected  not  to 
include  them  in  efforts  to  quantify  the  relationship  between  analyst  attributes  and  performance. 
First,  for  collaboration,  since  this  effort  was  focused  on  individual  activities  associated  with 
analysis,  the  ability  to  work  with  others  was  considered  outside  the  bounds  of  the  initially 
established  investigation.  However,  it  is  important  to  note  that  the  trend  towards  team-based 
analysis  for  intelligence  professionals  may  increase  the  importance  of  collaboration  for 
successful  product  generation.  As  such,  we  recommend  that  this  attribute  and  the  others 
identified  for  this  effort  be  reexamined  as  part  of  an  effort  to  understand  the  impact  of  team- 
based  analysis.  In  regard  to  organization  of  information,  the  team  suspects  that  this  ability  is 
likely  a  higher- level  construct  that  requires  the  analyst  to  have  a  number  of  lower- level  attributes 
(e.g.,  critical  thinking,  deductive  reasoning,  etc.)  to  effectively  perform. 
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Figure  1  shows  the  form  used  to  note  the  ratings  of  the  various  task  demands  for  each  of  the 
subtasks  of  the  Evaluate  the  Threat  task.  Table  E-2  reports  the  ratings  that  the  SMEs  gave  to  the 
attributes/abilities  relevant  to  the  subtasks  of  the  Evaluate  the  Threat  task. 


Task  # 

Activity  (or  Sub-Task) 

Ability  or  Task  Demand 

Needed  to  perform  the  activity: 

1  =  a  minimum  amount  is  needed 

4  =  a  moderate  amount  is  needed 

7  =  a  great  amount  is  needed 

1 

Memorizat 

ion 

2 

Selective 

Attention 

3 

Time 

Sharing 

4 

Deductive 

Reasoning 

5 

Inductive 

Reasoning 

6 

Problem 

Sensitivity 

7 

Flexibility 
of  Closure 

8 

Fluency  of 
Ideas 

9 

Originalit 

y 

10 

Persevera 

nee 

1 

Depict  the  composition  and  array  of  enemy 
network  in  an  AO,  Al,  and  AOI  based  on 
operational  variables  (PMESII-PT  ASCOPE)  and 

TTPs. 

2 

Develop  RFIs 

3 

Update  COP  with  current  enemy  composition 

4 

Develop  HVI/HVT/HPTs 

5 

Verify  gaps  in  intelligence  holdings 

6 

Identify  relevant  databases 

7 

Verify  METT-TC  effects  on  threat  forces 

8 

Draft  threat  assessments  to  develop  realistic 

Threat  models 

9 

Develop  enemy  OB/structure 

10 

Analyze  intelligence  holding  IOT  identify  existing 
or  emerging  enemy  TTPs 

11 

Develop  initial  ECOAs  based  upon 

12 

Develop  threat  capabilities  statement  in  Full 
Spectrum  is  focused  on  units 

Figure  1 .  SME  task  demand  rating  form. 

Table  4  presents  the  final  list  of  critical  aptitudes/abilities  needed  by  the  35F  all-source  analyst 
and  criticality  ratings  by  SMEs.  SMEs  1,  2,  and  3  graded  all  the  abilities  and  used  a  grading 
scale  of  A,  B,  C,  D,  and  F,  with  A  being  the  best;  SME  4  used  a  scale  of  1  to  7,  with  1  being  the 
highest  level  of  importance,  and  the  abilities  that  were  not  rated  still  being  considered  critical  but 
not  one  of  the  top  seven  abilities. 

Based  on  the  interview  results  and  associated  discussions  with  the  SMEs,  the  following  results 
were  obtained: 

•  The  nine  identified  cognitive  attributes  and  their  associated  definitions  (see  table  4, 
attributes  1-9)  were  confirmed  by  the  SMEs  to  be  relevant  for  35F  analysts;  however,  the 
SMEs  added  one  additional  attribute  referred  to  as  “Perseverance”  to  the  list  (see  attribute 
no.  10).  The  SMEs  subsequently  added  two  more  abilities/skills  to  the  list  of  critical 
attributes  for  the  35F  all-source  analyst,  Organization  of  Information  and  Collaboration, 
resulting  in  the  list  of  12  attributes  shown  in  table  4. 

•  It  was  decided  that  the  task  of  primary  interest  for  our  model  should  be  “Evaluate  the 
Threat”  as  it  met  the  preestablished  characteristics  of  requiring  cognitive-relevant 
components  of  analysis,  reasoning,  and  decision  making  and  was  identified  by  SMEs  as 
one  of  the  most  critical  35F  tasks. 
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•  The  task  element  characteristics  and  the  attribute  ratings  for  selected  task  elements  were 
obtained  and  translated  into  appropriate  coding  language  for  use  in  the  exploratory  model. 

Table  4.  Final  list  of  critical  aptitudes/abilities  needed  by  the  35F  all-source  analyst  and  criticality  ratings  by  SMEs. 


Ability 

Description 

SME  1 

SME  2 

SME  3 

SME  4 

1 

Memorization 

An  ability  to  remember  information,  such  as  words, 
numbers,  pictures,  and  procedures. 

B 

B 

A 

3 

2 

Selective 

attention 

An  ability  to  concentrate  on  a  task  over  a  period  of  time 
without  being  distracted. 

D 

A 

D 

— 

3 

Time-sharing 

An  ability  to  shift  back  and  forth  between  activities  or 
sources  of  information,  such  as  speech,  sounds,  touch, 
or  other  sources. 

C 

B 

C 

7 

4 

Deductive 

reasoning 

An  ability  to  apply  general  rules  to  specific  problems  to 
produce  answers  that  make  sense. 

A 

A 

A 

1 

5 

Inductive 

reasoning 

An  ability  to  combine  pieces  of  information  to  form 
general  rules  or  conclusions  (includes  finding  a 
relationship  among  seemingly  unrelated  events). 

A 

A 

A 

2 

6 

Problem 

sensitivity 

An  ability  to  tell  when  something  is  wrong  or  is  likely  to 
go  wrong.  It  does  not  involve  solving  the  problem,  only 
recognizing  that  there  is  a  problem. 

B 

B 

B 

5 

7 

Flexibility  of 
closure 

An  ability  to  identify  or  detect  a  known  pattern  (e.g.,  a 
figure,  object,  word,  or  sound)  that  is  hidden  in  other 
distracting  material. 

B 

A 

B 

8 

Fluency  of 
ideas 

An  ability  to  come  up  with  a  number  of  ideas  about  a 
given  topic  (the  number  of  ideas  is  important,  not  their 
quality,  correctness  or  creativity). 

D 

B 

B 

4 

9 

Originality 

An  ability  to  come  up  with  unusual  or  clever  ideas  about 
a  given  topic  or  situation,  or  to  develop  creative  ways  to 
solve  a  problem. 

C 

C 

C 

10 

Perseverance 

An  ability  to  want  to  coax  the  best  possible  answer  out 
of  a  situation  and  act  on  it  in  an  appropriately  timely 
manner  knowing  the  situation  and  associated 
repercussions. 

A 

C 

D 

6 

11 

Organization  of 
information 

An  ability  to  come  up  with  a  system  or  method  to  save 
and  store  information  for  retrieval  and  communication. 

C 

B 

B 

— 

12 

Collaboration 

An  ability  and  willingness  to  share  and  debate/discuss 
ideas  with  others. 

A 

A 

B 

— 

2.5  IMPRINT  Modeling 

IMPRINT*  is  a  simulation  and  modeling  tool  that  provides  a  means  to  estimate  Manpower, 
Personnel,  and  Training  (MPT)  and  Human  Factors  Engineering  requirements  and  to  identify 
constraints  for  new  weapon  systems  early  in  the  acquisition  process.  The  IMPRINT  tool  grew 
out  of  common  U.S.  Air  Force,  Navy,  and  Army  MPT  concerns  identified  in  the  mid-1970s.  It 
is  government-owned  software  and  consists  of  a  set  of  automated  aids  to  assist  analysts  in 


*http://www.arl.army.mil/www/default.cfm?page=445  (accessed  March  2012). 


21 


conducting  human  performance  analyses.  IMPRINT  has  been  available  as  a  government  product 
free  of  charge  since  the  mid-1990s  to  U.S.  government  agencies,  U.S.  private  industry  with  U.S. 
government  contract,  and  U.S.  colleges  and  universities  working  in  Human  System  Integration. 

It  is  supported  by  commercial-quality  user  documentation,  a  training  course,  and  a  technical 
support  organization.  Upgrades  and  enhancements  to  IMPRINT  have  been  driven  by  user 
requirements,  human  modeling  research,  and  changes  in  the  state  of  the  art  in  computer 
simulation.  IMPRINT  provides  a  powerful  and  flexible  environment  in  which  to  develop  human 
performance  models  and  has  unique  capabilities  for  assessing  the  impact  of  stressors  (e.g.,  noise, 
heat,  sleep  deprivation,  protective  gear)  on  performance. 

One  of  the  most  powerful  and  unique  capabilities  in  IMPRINT  is  the  method  through  which 
Soldier  characteristics  and  environmental  stressors  can  be  used  to  impact  task  performance.  This 
is  achieved  through  an  embedded  simulation  engine,  based  upon  the  commercial  Micro  Saint 
Sharp  product*  and  supplemented  by  human  performance  algorithms.  The  application  includes  a 
graphical  user  interface  (GUI)  shell  that  elicits  information  from  the  user  needed  to  assess  human 
performance  issues  associated  with  the  operations  and  maintenance  tasks  of  a  weapon  system. 
The  simulation  and  analysis  capabilities  in  IMPRINT  along  with  the  embedded  data  and  GUI 
have  been  demonstrated  to  enable  human  factors  professionals  to  impact  system  design  and 
acquisition  decisions  based  on  early  estimation  of  Soldiers’  abilities  to  operate,  maintain,  and 
support  the  system. 

Following  is  a  description  of  the  tasks  modeled,  how  data  to  support  and  populate  it  were 
collected,  and  the  scenarios  against  which  the  model  was  run  (i.e.,  Stability  and  Support 
Operations  [SSO],  Full-Spectrum  Operations  [FSO]). 

For  this  effort,  the  activities  to  be  modeled  were  those  involved  with  a  level-20  IA  (i.e.,  a  35F) 
performing  the  “Evaluate  the  Threat”  activity.  This  activity,  while  agreed  upon  as 
representative,  is  only  1  of  72  “activities”  an  IA  may  perform  and  only  1  of  18  that  a  level-20  IA 
may  be  called  upon  to  perform  (figure  2). 

Each  task  node  consists  of  a  single  task  (indicated  by  an  oval)  or  a  collection  of  subtasks 
(indicated  by  a  rectangle).  Within  each  rectangle  is  a  subnetwork  that  consists  of  tasks  and 
possibly  more  sublevels  of  subnetworks.  For  example,  the  rectangle  depicting  the  task  of 
Evaluate  the  Threat  for  a  skill  level-20  analyst  (task  301-35F-2253)  is  shown  in  figure  3.  (The 
tasks  conducted  by  an  IA  with  a  skill  level  of  20,  according  to  the  PS&M,  are  listed  in  table  5.) 
This  subnetwork  comprises  the  tasks  that  constitute  what  a  level-20  analyst  should  be  able  to 
perform.  The  ratings  that  the  SMEs  gave  to  the  attributes/abilities  relevant  to  the  subtasks  of  this 
activity  are  given  in  table  D-l. 

Each  task  node  consists  of  arithmetic  logic  and  algorithms  that  are  used  to  model  attributes  of 
interest.  For  example,  each  node  encapsulates  an  algorithmic  expression  of  task  time  that 


*http://www. alionscience.com/Technologies/Simulation-and-Visualization/Micro-Saint-Sharp  (accessed  March  2012). 
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represents  the  duration.  Further,  upon  executing  the  task,  logical  expressions  and  equations  can 
be  evaluated  and  calculated,  respectively,  to  model  interactions  between  tasks  and  between  tasks 
and  the  operational  environment.  For  the  purposes  of  this  effort,  each  task  calculated  the  amount 
of  time  in  hours  per  12-h  shift  that  the  analyst  would  perform  this  subtask.  Software  code  to 
implement  this  task  time  is  written  in  the  programming  language  C#  (pronounced  C  sharp),  an 
example  of  which  is  shown  in  figure  4. 

Similarly,  upon  the  execution  of  each  task,  the  code  within  the  task  is  activated.  The  code  shown 
in  figure  5  is  used  to  calculate  a  “score”  for  each  cognitive  ability  (or  task  demand)  and  is  the 
summation  across  all  the  subtasks  of  the  task  time  duration  (in  units  of  hours  per  12-h  shift)  and 
a  rating  offset.  The  rating  offset  is  a  compilation  of  the  ratings  given  by  the  SMEs  to  each 
cognitive  ability. 
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Figure  2.  Task  network  diagram  of  the  tasks  performed  by  a  35F. 
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[  Produce  a  Target  Folder  s 
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48Task30l-35F-2201  ( 
Ve-ify  Ta-ge:  Folder 


9Task301-35F-1251 
Define  Operational  Environment 

Condition:  Giver  a  warning  order  {WARNO}.  hgher  echetor  intelligence 
( estmate.  current  intelligence  holdings.  FM2-0.  FM2-01.3.  and  mission 
req  urements . 

Standards  Define  the  operational  environment  in  terms  of  identification  of  the 
a *ea  of  operators  {AO),  sgrrear:  craracterstcs  of  the  environment,  and 
~:e  ge~:eg=:;  -  wt~  -M  I-i'  2 


81  Task  301 -35F-2251 
Define  Operational  Environment 

Condition:  Given  a  warring  order  {WARNO}.  higher  echelon  intelligence 
estmate.  current  intelligence  holdings.  FM  2-0.  FM  2-01 .3.  and  mission 
1  requirements 

Standards :  Develop  the  definition  of  the  operational  environment  in  terms  of 
determining  the  a*e3S  of  operations  {AO}  anc  areas  of  interest  (Al).  sgnifcart 
characteristics  of  the  environment,  ard  mtellgence  gaps  in  accordance  with 
FM  2-01.3. 


68  Task  301 -35F -3251 

Define  the  Operational  Environment 

Condition:  Given  a  wamng  order  (WARNO).  higher  echeton  rtefcgerce 
estimate,  current  mergence  holdings.  FM  2-0.  FM  2-01.3.  and  meson 
requirements. 

Standards  Venfy  the  definition  of  the  operational  envronment  ri  terms  of 
identification  of  the  areas  of  operations  (AOs)  and  areas  of  rterest  (Als), 
significant  characteristics  of  the  environment,  rtefcgerce  gaps  and  Mate 
coiection  of  nformatior  requred  n  accordance  w*h  FM  2-01  3 


80***Task301-35F-1252 

Describe  environmental  effects  on  threat  and  friendly  operations 

Condition  Gwen  current  weather  reports  area  of  rterest  (Al).  a  situational  map. 
warning  order  (WARNO).  araiyss  and  products  from  rtellgence  preparation  of 
the  battlefield  {IPS}  step  1 

Standards :  Describe  environmental  effects  or  threat  and  fnendly  forces  m  terms 
of  draftirg  terrain  araiyss.  drafting  analyse  of  military  aspects  and  effects  of 
terram .  while  applying  the  effects  of  weather  on  miltary  operations .  and 
considering  crviliar  factors  of  a-ea.  structures .  capabt  t  es  orgarczators . 
people,  and  events  (ASCOPE)  m  accordance  with  FM  2-01  3. 


8 ‘"Task  301-35F-2252 

Describe  environmental  effects  or  threat  and  fnendly 

Condition:  Gwen  current  weather  reports,  area  of  interest  (Al).  a  skuatoral  map 
(SITMAF).  warrmg  order  (WARNO).  analysis  and  products  from  intelligence 
preparation  of  the  battlefield  (IPB)  step  1 

Standards  Develop  the  descriptor  of  ervirormental  effects  on  threat  ard 
fnendly  forces  m  terms  of  conducting  terram  araiyss.  developing  analyss  of 
military  aspects  ard  effects  of  terram,  while  applyrg  the  effects  of  weather  on 
military  operations,  and  corsdering  civilian  factors  of  area,  structures, 
capabilities,  organizations,  people,  a-c  events  {ASCOPE}  in  accordance  with 
FM  2-01 .3. 


67 —Task  301-35F -3252 

Describe  envronmemal  effects  on  threat  and  fnendly 

Condnon:  Giver  current  weather  reports,  area  of  rterest  (Al).  a  scuatonaJ  map 
(SITMAP).  warn  mg  order  (WARNO).  analyss  and  products  from  rtefcgerce 
preparation  of  the  battlefield  (IPB)  Step  1 

Standards :  Verify  the  description  of  envrormertal  effects  on  the  threat  and 
fnendly  forces  r  terms  of  verifyrg  teoar  analyss.  analyzrg  m*tary  aspects  and 
effects  of  terram .  whie  applyrg  the  effects  of  weather  on  mitary  operations . 
and  considering  cwfcan  factors  (ASCOPE)  in  accordance  wth  FM  2-01 .3. 


S3Task301-25F-1253 

1 0  Tas  k  301 -35F-2253 

82  Task  301 -35F -3253 

Evaijate  the  T  hreat 

Evaluate  the  Threat 

Evaluate  the  Threat 

Condition:  Gwen  a  warn  mg  order  (WARNO).  hgher  echetor  mtellgence 

Conditor.  Gwen  a  warning  order  (WARNO).  hgrer  ecnetor  mtellgence 

Cordoon  Gwen  a  wamng  order  (WARNO).  higher  echeton  mergence 

,  estmate.  current  rtefcgerce  holdings.  FM  2-0.  FM  2-01.3.  mssor 

,  estmate.  current  mtellgence  holdings.  FM  2-0.  FM  2-01 .3.  ard  mssor 

estmate.  current  rtefcgerce  holdmgs.  FM  2-0.  FM  2-01  3.  and  mssor 

requirements ,  ard  products  from  mtellgence  preparator  of  the  battlefield  (IPB) 

-egu  -emerts  ard  products  from  irteUgerce  preparator  cf  the  battlefield  (IPB) 

requirements  and  products  from  rtefcgerce  preparation  of  the  battlefield  (IPB) 

steps  1  and  2. 

steps  1  and  2. 

steps  1  and  2. 

Standards :  Evaluate  the  threat  m  terms  of  drafting  threat  capabilities  whie 

Standards :  Evaluate  the  threat  m  terms  of  devetopmg  threat  capabilities  wh  ile 

Standards  Evaluate  the  threat  r  terms  of  devetopng  threat  capacities  w hue 

drafting  and  updatmg  threat  models  m  accordance  with  FM  2-01 .3. 

creating  and  updatmg  threat  models  m  accordance  with  FM  2-01 .3. 

creating  and  updatmg  threat  models  r  accordance  with  FM  2-01 .3. 

51  Task  301 -35F-2254 

Determ  me  threat  Cojrses  o#  Acton 

Condition:  Gwen  a  warrmg  order  (WARNO),  hgher  echeton  rtellgence 
estmate.  current  rteiigerceholdings.  FM2-0.  FM-01.3.  mssor  requirements 
ard  products  from  rtefcgerce  preparation  of  the  battlefield  (IPB)  steps  1 . 2.  ard  3. 

Standards  Determ  me  threat  courses  of  action  (COAs)  in  terms  of  develop  -g 
threat  objectives  and  (COAs).  devetopmg  enemy  situation  templates  ard  high 
value  target  fasts  (HVTLs).  ard  devetopmg  mitial  rteflgerce.  su-ve:3ance.  and 
reconnassarce  { ISR)  -equiremerts  r  accordance  with  FM  2-01 .3. 


6Task301-35F-l254 
Develop  the  Situation 

Condition  Gven  an  operation  order  (OPORD).  initial  rtellgence  preparator  of 
the  battlefield  (IPB)  products,  ard  orgoirg  m  jlh-dscjjtire  rtellgence  co^ecton 
t  operators  in  ar  a-ea  cf  operators  (AO)  or  area  cf  interest  (Al). 

Standards:  Perform  skuaton  development  by  receiving,  processmg.  and 
aralyzirg  inform auon  m  ordertodete-mme  inteCgence  gaps:  confirm  or  deny 
threat  courses  of  acton  (COAs);  update  situation  map  (SITMAP).  and  draft 
mtetlgercesjmmary  (INTSUM)  m  accordance  with  FM2-0.  FM2-01.3.  FM 
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Figure  2.  Task  network  diagram  of  the  tasks  performed  by  a  35F  (continued). 


28  Task  301-35F-1256 

Provide  irtelligerce  support  to  targetirg 

Condition  Giver  the  c  jrrem  situation  in  a  selected  operational  environment  FM 
6-20.10  (FM  3-60),  JP3-0.  MCRP3-1.6.14.  FM  7-15.  s«uatonmap(SITMAP). 

FM  2-0.  FM  2-01.3.  MCRP2-3A.  meson  requirements.  threat  modei.  message 
traffic,  commander's  intent, eremy  order  of  battle  {OB)charts.  mission  statement, 
and  pnonty  irtelligerce  requirements  (PIRs). 

Stardards  Provide  irtelligerce  support  to  targetirg  by  ider.tifyirg  potertial  high 
value  target  {HVT}  arc  irtelligerce  gaps  Draft  a  target  rteliigerce  packet  {TIP) 
ir  accordance  wsh  FM  2-01  .3  ard  FM  2-0 


Condeior :  Giver  the  cjrrent  sr.uaton  r  a  giver  operatoral  ervrorvnert.  FM 
6-20-10 (FM  3-60).  JP3-0,  MCRP 3-1. 6.14.  FM  7-15.  situation  map(SITMAP). 
FM2-0.  FM  2-01.3.  MCRP2-3A.  mission  requirements,  threat  model,  message 
traffic  commander's  intent  mssor  statement,  pnonty  rteliigerce  requireme-ts 
(PIRs).  ard  irtellgerce  preparator  of  the  battlefield  (IPS}  targeting  products. 

Standards  Provide  irtelligerce  support  to  targeting  by  developing  targets .  high 
value  target  list  (HVTL).  ard  target  rteliigerce  packet  (TIP).  Validate 
irtellgerce  gaps  Aralyze  ard  prioritize  targets  tr  accordarce  with  FM  2-0  and 
FM  2-01 .3. 
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78  Task  301-35F-4257 
I  integrate  irtelligerce  s  upport  to  targetirg 

Condition:  Giver  the  current  situation  in  a  selected  operational  environment.  FM 
6-20-10,  JP3-0.  MCRP  3-1  6  14.  FM  7-15.  SITMAP.  FM2-01  3.  MCRP2-3A. 
mission  requirements,  threatmodel.  message  traffic,  commander's  intent,  mission 
statement,  and  priority  intelligence  requirements  (PIRs). 

Standards:  Integrate  intelligence,  surveitence,  ard  reconnaissance  (ISR).  ard 
multi-source  intelligence:  recommend  targetirg  effects.  Recommerd  re-engage 
by  iethal  or  ror-lethal  means  ir  accordarce  with  FM  2-0  and  FM  2-01 .3. 


,7Task301-35F-1404 
Marta:'-  theSituator  Map  (SITMAP) 


54  Task  301-35F -2351 

Verfy  theSituator  Map  (SITMAP) 
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35  Task  301 -35F-1 407 
Write  an  Analytical  Paper 
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I  41  Task  301-35F-1602  J 

1  Conduct  Automated  I  rteliigerce  System  (AIS)  Operations  | 


57  Task  301 -35F-2458 

Supervse  Automated  Intelligence  System  (AIS)  Operators 


Figure  2.  Task  network  diagram  of  the  tasks  performed  by  a  35F  (continued). 
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Figure  3.  Task  network  diagram  of  Evaluate  the  Threat,  skill  level-20  subtasks. 


Table  5.  Level-20  subtasks  of  Evaluate  the  Threat. 


Task 

No. 

Activity  (or  Subtask) 

1 

Depict  the  composition  and  array  of  enemy  network  in  an  area  of  operation  (AO)  and  area  of 
interest  (AI  or  AOI)  based  on  operational  variables  (PMESII-PT  ASCOPE)  and  tactics, 
techniques,  and  procedures  (TTPs). 

2 

Develop  requests  for  information  (RFIs). 

3 

Update  the  common  operational  picture  (COP)  with  current  enemy  composition. 

4 

Develop  high-value  individual  (HVI)/high-value  target  (HVT)/high-payolT  targets  (HPTs). 

5 

Verify  gaps  in  intelligence  holdings. 

6 

Identify  relevant  databases. 

7 

Verify  METT-TC  effects  on  threat  forces. 

8 

Draft  threat  assessments  to  develop  realistic  threat  models. 

9 

Develop  enemy  order  of  battle  (OB)/structure. 

10 

Analyze  intelligence  holding;  identify  existing  or  emerging  enemy  TTPs. 

11 

Develop  initial  enemy  courses  of  action  (ECOAs)  based  upon  indicators  and  historical  TTPs. 

12 

Develop  threat  capabilities  statement. 

//  task  duration  is  in  hrs/12-hr  shift 

//  In  a  12-hr  shift.  Indicate  the  #  of  hours  spent  doing  this 
double  taskDuration  =  0.0; 
double  numShifts  =  1.0; 

if  (IA[Entity.Tag] . TypeOfOperation  ==  OPERATION_FSO) 

{ 

//  1-2  hrs,  every  12-24  hrs  (every  1  to  2  12-hr  shifts) 
taskDuration  =  Distributions . Rectangular ( 1 . 5 ,  1.0); 
numShifts  =  Distributions . Rectangular ( 1 . 5 ,  1.0); 

} 

else  if  ( IA [Entity . Tag] . TypeOfOperation  ==  OPERATION_SSO) 

{ 

//  10-15  min,  ongoing  (0  to  4  times  per  12-hr  shift) 
taskDuration  =  Distributions . Rectangular ( 15 . 0 ,  10.0)  /  60.0; 

numShifts  =  Distributions . Rectangular (2 . 0 ,  0.0); 

} 

else 

return  0.0; 

return  taskDuration  /  numShifts; 


Figure  4.  Sample  model  code  for  task  time  calculation. 
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//  Score  =  time  duration  in  hours  *  cognitive  ability  rating 
//  given  by  the  SMEs  to  this  ability  for  this  task. 

//  Collect  a  scoring  for  each  of  the  following: 

//  skill  level 
//  type  of  operation 

//  cognitive  ability 

double  normalizedTaskRating; 
double  ratingOffset  =  0.0; 

for  (int  thisAbility=0 ;  thisAbility<NumAbilities ;  thisAbility++) 

{ 

//  What  is  the  difference  between  a  "good"  analyst  and  this 

guy? 

normalizedTaskRating  = 

(TaskRatingOfGoodAnalyst [thisAbility]  - 
1.0)  /  (double) (7  -  1); 

ratingOffset  =  normalizedTaskRating  - 
IA [Entity . Tag] . AbilityRating [thisAbility] ; 

IA [Entity . Tag] . ETT_Score [thisAbility]  +=  (Entity . Duration  * 
ratingOffset) ; 

//  Model . PrintOutput ( "Entity  "  +  Entity. Tag 
//  +  "  this  Ability  "  +  thisAbility 

//  +  "  normalizedTaskRating  "  +  normalizedTaskRating 

//  +  "  IA [Entity . Tag] .AbilityRating [thisAbility]  "  + 

IA [Entity. Tag] .AbilityRating [thisAbility] 

//  +  "  Entity . Duration  "  +  Entity . Duration) ; 

} 


Figure  5.  Sample  model  code  for  attribute  ratings. 


3.  Results 


3.1  Cognitive  Attribute  Effects  on  Performance 

The  purpose  of  intelligence  analysis  is  to  create  intelligence  information  from  data  acquired  from 
a  variety  of  sources.  A  successful  analyst  produces  an  intelligence  product  that  conveys 
intelligence  information  and  meets  the  consumer’s  needs.  Performing  the  IA  job  requires 
appropriate  skills  and  abilities,  knowledge,  and  personal  characteristics  that  are  associated  with 
rigorous  intelligence  analysis  and  production.  The  ability  to  communicate,  cooperate,  and  think 
critically,  coupled  with  the  skills  that  ensure  technical  competency,  is  a  requirement  for 
intelligence  work.  Knowledge  of  the  issues  and  their  background  provides  both  content  and 
context  for  analysis.  “Analysts  who  are  motivated  to  succeed,  to  know  targets,  and  to  share  that 
knowledge  ensure  that  consumers  receive  intelligence  of  the  highest  caliber”  (Swenson,  2003). 
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The  goal  of  this  effort  was  to  identify  the  characteristics  that  an  all-source  analyst  must  possess 
to  succeed  in  these  activities.  The  work  conducted  to  achieve  this  goal  produced  a  variety  of 
outputs  that  have  value  to  the  Army  intelligence  community  and  the  intelligence  community  at 
large.  Specifically,  the  standardized  test  available  for  measuring  the  attributes  needed  for  a 
successful  analyst  provides  a  way  for  improving  the  selection  and  assignment  of  individuals  to 
the  IA  job  family.  Additionally,  the  quantitative  relationships  between  analyst  attributes  and 
performance  established  through  the  meta-analytic  activities  can  aid  in  prioritizing  the  extensive 
set  of  required  attributes  based  on  their  likely  contributions  to  overall  performance,  while  the 
gaps  identified  in  the  literature  search  suggest  paths  forward  for  rounding  out  the  theory-driven 
investigations  needed  to  develop  a  more  comprehensive  approach  to  selection,  assignment,  and 
training  of  intelligence  professionals.  Finally,  the  skill  usage  information  generated  from  the 
task  network  modeling  efforts,  in  conjunction  with  the  test  information  and  performance 
relationships,  can  point  to  high-priority  attributes  that  are  likely  to  be  utilized  most  often  across 
intelligence  analysis  mission  types  and  as  such  have  more  significant  impacts  on  overall 
performance. 

The  following  sections  summarize  the  outputs  of  the  investigations  conducted  under  this  effort, 
and  the  report  concludes  with  a  set  of  recommendations  that  are  intended  to  provide  a  basis  for 
continued  investigation  and  actions  that  will  improve  the  performance  of  IAs  for  the  U.S.  Army. 

3.1.1  Standardized  Tests 

Appendix  C  provides  a  summary  of  the  tests  identified  from  the  literature  and  from  the  Mental 
Measurement  Yearbook  (Spies  et  al.,  2010)  that  measure  attributes  considered  by  researchers  and 
SMEs  to  be  critical  to  IA  performance.  Each  test  identified  could  improve  the  ability  to 
select/assign  personnel  to  an  intelligence  analysis  MOS  by  adding  a  predictor  to  the  overall 
assessment  battery  used  for  MOS  assignment.  The  assessments  vary  in  the  resource  demands  for 
administration,  but  as  noted  in  the  recommendations,  implementation  of  targeted  attribute  tests 
from  those  identified  here  may  offer  some  near-term  improvements  to  IA  performance. 

3.1.2  Performance  Relationships 

Appendix  D  provides  a  summary  of  performance  effects  compiled  from  the  literature.  It 
contains  descriptions  of  effects  matrices  that  target  one  or  more  of  the  cognitive  abilities.  From 
this  information,  the  following  high-level  results  emerge: 

•  Critical  thinking  was  found  to  impact  the  broadest  range  of  task  demand  groups  of  the  13 
measureable  attributes  reviewed  for  this  effort. 

•  Flexibility  of  closure  appeared  in  more  quantitative  studies  than  any  other  attribute,  and  as 
such,  its  impact  on  performance  in  IA-related  tasks  is  well  established. 

•  Tasks  requiring  deductive  reasoning,  inductive  reasoning,  and  flexibility  of  closure  are 
frequently  used  in  the  experimental  literature;  the  impact  of  personnel  attributes  on  those 
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tasks  is  well  established  and  may  be  a  useful  starting  point  in  improving  analyst  task 
performance. 

•  Although  only  found  in  one  study,  Motivation/Perseverance  appears  to  have  a  strong 
performance  impact  on  tasks  requiring  collaboration. 

3.1.3  Gaps/Missing  Pieces  in  the  Literature 

The  database  searches  that  attempted  to  identify  empirical  research  linking  attribute  levels  to  task 
performance,  based  on  the  tests  used  to  measure  attribute  levels,  identified  tens  of  thousands  of 
abstracts  with  potential  data  linking  personnel  attributes  to  task  performance.  Time  and  resource 
limitations  on  the  current  effort  only  allowed  for  inclusion  of  several  hundred  of  these  studies  as 
part  of  the  current  meta-analysis  effort.  Consequently,  further  exhaustive  data  collection  could 
reveal  that  some  of  the  gaps  identified  in  the  current  matrix  are  the  result  of  sampling  rather  than 
a  true  gap  in  the  literature.  Additionally,  further  work  would  be  needed  to  increase  the  certainty 
around  the  estimated  effects  noted  here.  However,  based  on  this  cursory  review,  a  few  likely 
gaps  can  be  hypothesized  based  on  the  relatively  low  numbers  of  studies  included  in  our  initial 
sampling.  For  instance,  our  results  found  only  one  study  with  quantitative  impacts  of  induction 
on  task  performance.  Likewise,  only  one  study  quantifying  the  impacts  of  motivation 
/perseverance  on  task  performance  was  identified.  These  attributes,  along  with  some  specific 
attribute-to-task  demand  pairings,  may  bear  further  investigation  in  the  literature  to  establish 
them  as  true  gaps  in  knowledge  and  to  determine  the  likely  payoff  in  terms  of  improved 
predictive  ability  that  could  be  gained  if  the  gaps  were  closed. 

3.2  Modeling  Results:  Skill  Usage  Profiles  Across  Mission  Types 

Model  results  were  calculated  for  FSO  (table  6)  and  SSO  (table  6).  Based  on  information 
gathered  from  IA  SMEs,  the  amount  of  time  spent  on  each  skill  level-20  subtask  of  the  Evaluate 
the  Threat  task  was  estimated  on  a  yearly  basis  (table  E-2).  These  hours  were  then  summed  for 
each  of  the  10  task  demands  for  each  rating  level.  Again,  the  rating  levels  were  obtained  from 
SME  interviews. 

When  looking  at  the  amount  of  time  the  analyst  will  spend  exercising  the  attributes  required  for 
task  performance  across  both  FSO  and  SSO,  we  see  from  the  model  that  each  attribute  is 
required  for  approximately  equal  amounts  of  time.  This  finding  reinforces  the  message  that  the 
SMEs  communicated  during  data  collection — that  all  the  identified  attributes  contribute  to 
successful  job  performance  for  the  all-source  analyst.  However,  when  the  time  spent  performing 
specific  tasks  is  examined  in  combination  with  the  attribute  criticality  ratings  provided  by  the 
SMEs,  a  few  attributes  appear  to  be  required  more  often  at  critical  levels.  For  FSO,  the  attributes 
of  time  sharing,  problem  sensitivity,  and  flexibility  of  closure  have  more  time  of  utilization  on 
tasks  where  those  attributes  are  needed  most  for  task  completion.  Similarly,  in  SSO,  time 
sharing,  problem  sensitivity,  and  motivation/perseverance  are  used  for  the  longest  duration  on 
tasks  where  those  attributes  are  in  high  demand. 
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Table  6.  Hours  per  year  spent  utilizing  specific  abilities  for  a  level-20  IA  while  conducting  Evaluate  the  Threat 
during  full-spectrum  operations. 


Rating 

Memorizati 

on 

Selective 

Attention 

Time¬ 

sharing 

Deductive 

Reasoning 

Inductive 

Reasoning 

Problem 

Sensitivity 

Flexibility 

of  Closure 

Fluency  of 

Ideas 

Creativity  / 
Originality 

Motivation 

/ 

Perseveran 

ce 

i 

13 

13 

2 

13 

3 

13 

836 

3  1/2 

13 

4 

35 

1,166 

41/3 

41/2 

35 

35 

4  2/3 

5 

114 

814 

78 

8 

56 

91 

92 

5  1/3 

5  1/2 

178 

365 

36 

78 

5  3/4 

36 

6 

801 

460 

13 

836 

122 

78 

514 

79 

78 

6  1/4 

1 

814 

801 

6  1/3 

6  1/2 

37 

365 

365 

464 

6  2/3 

6  3/4 

87 

7 

186 

91 

1,389 

91 

925 

893 

81 

An  additional  output  from  the  modeling  effort  that  can  help  focus  future  actions  for  improving 
overall  analyst  performance  is  results  indicating  the  amount  of  time  an  analyst  spends  on  specific 
tasks  over  the  course  of  the  year.  When  the  Evaluate  the  Threat  tasks  are  simulated  for  over  a 
year  of  performance,  the  model  can  point  to  high  driver  tasks  that  can  be  expected  to  have 
significant  impacts  on  mission  performance.  For  each  high  driver  task,  a  set  of  required 
cognitive  attributes  is  identified  that,  along  with  the  results  in  tables  6  and  7,  can  help  focus 
future  action  on  attributes  that  are  likely  to  have  the  most  impact  on  overall  performance. 

On  the  whole,  the  model  outputs  suggest  that  the  IA  should  possess  the  diverse  set  of 
characteristics  suggested  by  the  literature  search  and  confirmed  by  the  SMEs.  Additionally,  the 
model  results  taken  in  conjunction  with  the  SME  ratings  may  indicate  that  time  sharing  and 
problem  sensitivity  could  be  considered  more  important  for  the  analyst,  while  attributes  such  as 
creativity,  originality,  and  fluency  of  ideas — while  frequently  exercised  in  performing  IA  tasks — 
are  required  to  a  lesser  degree  for  mission  performance  than  the  other  attributes  identified. 
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Table  7.  Hours  per  year  spent  utilizing  specific  abilities  for  a  level-20  IA  while  evaluating  the  threat  during  stability 
and  support  operations. 


Rating 

Memorizati 

on 

Selective 

Attention 

Time¬ 

sharing 

Deductive 

Reasoning 

Inductive 

Reasoning 

Problem 

Sensitivity 

Flexibility 
of  Closure 

Fluency  of 
Ideas 

Creativity  / 
Originality 

Motivation 

/ 

Perseveran 

ce 

1 

13 

13 

2 

13 

3 

13 

126 

31/2 

13 

4 

35 

91 

41/3 

78 

91 

41/2 

35 

35 

4  2/3 

104 

91 

78 

5 

2,268 

104 

78 

8 

56 

195 

351 

51/3 

35 

104 

51/2 

299 

2,190 

78 

5  3/4 

260 

2,190 

6 

195 

476 

13 

230 

2,276 

78 

2,303 

598 

78 

61/4 

260 

104 

91 

61/3 

260 

104 

260 

61/2 

2,450 

104 

463 

6  2/3 

2,190 

260 

6  3/4 

208 

7 

567 

91 

3,213 

91 

3,009 

547 

2,615 

4.  Recommendations 


The  extensive  literature  review  and  modeling  effort  that  were  performed  revealed  a  great  deal 
about  the  requirements  for  effective  intelligence  analysis.  A  consensus  from  the  literature  exists 
that  the  duties  of  the  IA  are  diverse  and  complex,  requiring  a  variety  of  core  functional 
competencies  for  successful  completion.  Complex  relationships  exist  among  these  core 
competencies  required  for  effective  analysis,  and  more  specifically,  among  the  cognitive 
attributes.  These  complex  relationships  are  not  well  described  or  understood  in  the  literature. 
While  it  is  apparent  that  many  of  the  cognitive  attributes  that  are  described  as  characteristic  of  an 
effective  analyst,  both  in  the  literature  and  confirmed  as  part  of  this  effort,  are  conceptually 
related  to  one  another,  the  impact  that  the  individual  attributes  have  on  performance  and  the 
relationships  between  them  is  much  less  clear  or  well  understood.  This  fact  made  quantifying 
the  impact  of  varying  attribute  levels  on  mission  performance  a  significant  challenge.  However, 
the  human  performance  model  built  for  this  effort  highlights  the  attributes  that  are  important  over 
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time  in  an  analyst’s  mission  and,  as  such,  is  useful  in  directing  further  investigation  into  the 
relationships  among  attributes  and  between  attributes  and  ultimately  analyst  performance.  The 
literature  review,  SME  interactions,  and  model  development  and  analysis  efforts  have  pointed  to 
a  number  of  recommended  actions  that  could  be  undertaken  to  improve  IA  performance  either 
through  selection  and  assignment,  training,  or  capability  development. 

4.1  Immediate  Term 

Consider  implementing  a  test  for  Flexibility  of  Closure  as  part  of  the  analyst  selection  and 
assignment  process. 

Flexibility  of  Closure,  an  important  measureable  ability  that  seems  to  exhibit  strong  positive 
relationships  with  IA-like  tasks,  is  defined  as  follows: 

•  The  ability  to  identify  or  detect  a  known  pattern  defined  as  a  figure,  object,  word,  or  sound 
that  is  hidden  in  other  distracting  material. 

•  The  ability  to  demonstrate  selective  attention  to  a  specified  set  of  elements  when  presented 
within  various  settings  (the  larger  the  number  of  settings  from  which  the  desired  set  of 
elements  can  be  selected,  the  higher  the  level  of  flexibility  of  closure). 

•  The  ability  to  dissemble  information  from  a  distracting  field. 

Flexibility  of  Closure  is  something  readily  measureable  with  closure-specific  tests  or  closure 
indices  within  larger  test  batteries.  It  would  be  a  worthwhile  pursuit  to  test  current  IAs  on 
Flexibility  of  Closure  ability  and  assess  whether  scores  demonstrate  potential  as  a  descriptive 
discriminator  between  IA  that  have  “the  right  stuff’  and  those  that  do  not.  Alternately,  some 
researchers  assert  that  this  ability  can  be  improved  through  training.  Effort  should  be  undertaken 
to  determine  if  the  addition  of  training  materials  specifically  geared  to  improving  this  attribute 
would  support  improved  analyst  performance. 

Determine  if  existing  measures  of  critical  thinking  and  reasoning  skills  discriminate  between 
high-  and  low-performing  analysts  and  consider  introducing  such  an  assessment  to  existing 
selection  batteries. 

Critical  thinking  and  reasoning  skills  also  seem  to  exhibit  strong  positive  relationships  with 
IA-like  tasks.  In  particular,  the  model  results  indicate  that  analysts  spend  approximately 
one-third  of  their  time  developing  HVIs,  a  task  that  requires  deductive  reasoning  and  problem 
sensitivity.  The  meta-analysis  results  indicate  that  measures  of  critical  thinking  are  strong 
predictors  of  performance  on  these  types  of  tasks.  Because  of  the  amount  of  time  spent  on  this 
task  and  the  ability  of  critical  thinking  measures  to  predict  performance,  selecting  for  improved 
critical  thinking  skills  could  improve  performance  on  high-demand  tasks  like  developing  HVIs. 
An  immediate  benefit  could  be  realized  through  adding  a  validated  test  of  critical  thinking  skills 
to  the  selection  and  assignment  battery  for  all  source  analysts.  A  first  step  in  this  process  would 
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be  to  test  current  analysts  on  critical  thinking  and  reasoning  skills  and  compare  those  results  to 
performance  evaluations  to  determine  whether  scores  on  the  critical  thinking  test  demonstrate 
potential  as  a  descriptive  discriminator  between  high-  and  low-performing  analysts. 

Evaluate  the  potential  for  improving  deductive  reasoning  in  existing  analysts  as  a  means  for 
targeted  performance  improvement. 

The  model  results  suggest  that  two  of  the  activities  analysts  spend  the  most  time  performing, 
Develop  HVI  and  Develop  Enemy  Order  of  Battle,  both  require  moderate  to  high  levels  of 
deductive  reasoning.  Training  to  improve  deductive  reasoning  has  long  been  addressed  in  the 
research  literature  (see  Sleeman,  1975,  for  example),  and  the  capability  to  improve  deductive 
reasoning  ability  may  be  easily  available.  Because  of  the  frequency  of  use  and  relative 
importance  of  this  ability,  a  near-term  investment  in  efforts  to  improve  current  analyst  deductive 
reasoning  ability  may  provide  a  reasonable  return  on  investment. 

Further  explore  the  role  of  strategy  development;  executive  IP  functions  may  be  warranted. 

We  found  no  JCAT  attributes  that  reflected  what  we  refer  to  as  executive  IP  functioning  related 
to  strategy  development  and  use  that  are  often  associated  with  metacognitive  processes; 
therefore,  we  believe  that  the  exploratory  model  provided  for  this  project  may  be  expanded  to 
include  these  other  potentially  useful  cognitive  constructs,  assuming  that  future  empirical  results 
can  be  used  to  match  these  attributes  to  specific  IA  tasks. 

Identify  specific  quantifiable  performance  metrics  for  evaluating  analyst  performance  on  critical 
analysis  tasks. 

The  literature  review  found  that  the  lack  of  available  quantitative  measures  of  effective 
performance  on  IA  tasks  is  a  frequently  cited  challenge  in  efforts  to  understand  and  improve 
analyst  performance.  This  was  confirmed  in  the  current  effort  through  our  interactions  with 
analyst  SMEs  in  our  attempts  to  develop  quantitative  task-level  performance  criteria  for  use  in 
developing  algorithms  to  characterize  the  relationship  between  attribute  levels  and  analyst 
performance.  Identifying  and  describing  analysis  products  with  measures  of  performance  and 
effectiveness  that  can  be  quantified  could  provide  a  valuable  step  in  understanding  how  to 
predict  which  individuals  will  perform  well  as  IAs. 

4.2  Mid-Term 

Further  explore  attributes  of  successful  analysts  who  increasingly  function  in  a  team-based 
environment. 

With  increasing  amounts  of  information  available  and  the  ability  to  share  that  information  across 
organizational  boundaries  (e.g.,  between  units,  military  branches,  and  non-military  agencies), 
there  is  a  trend  toward  increased  team-based  analysis  activity  for  intelligence-related  data.  This 
effort  found  that  collaboration  is  an  important  characteristic  for  analysts  in  this  environment. 
However,  beyond  collaboration  requirements,  team-based  analysis  may  require  a  different  set  of 
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attributes  altogether,  and  understanding  the  differences  between  the  attributes  required  for 
individual  analytic  activity  vs.  team-based  activities  may  be  important.  Additionally,  questions 
such  as,  Can  one  team  member  with  above-average  abilities  in  a  specific  area  compensate  for  the 
lack  of  that  ability  in  another  team  member,  are  likely  to  become  more  relevant  as  analysis 
continues  to  develop  into  a  team-level  activity. 

More  broadly  examine  the  range  of  ability  tests  that  can  discriminate  good  from  bad  analysts 
above  and  beyond  the  current  Armed  Services  Vocational  Aptitude  Battery  (ASVAB)  test 
composites  that  are  used  for  selection  into  the  specialization. 

As  outlined  in  appendix  C  of  this  report,  a  number  of  validated  and  generally  available 
instruments  exist  for  measuring  the  cognitive  attributes  required  for  successful  IAs.  Many  of 
these  assessments  may  improve  the  Army’s  ability  to  select  individuals  into  an  intelligence 
MOS.  In  order  to  understand  the  potential  improvements  in  predictive  ability  that  could  be 
provided  by  a  subset  of  these  assessments,  a  study  could  be  conducted  to  evaluate  the  added 
predictive  power  that  selected  tests  may  provide  above  and  beyond  what  is  currently  provided  by 
the  ASVAB. 

Conduct  controlled  experiments  and  testing  to  identify  the  breadth  of  ability  tests  that  prove 
useful  for  discriminating  good  from  bad  analysts  and  predict  success  (and  therefore  informing 
selection )  across  missions. 

Because  the  job  of  an  all-source  analyst  is  complex  and  requires  a  variety  of  higher-level 
cognitive  abilities,  the  Army  would  benefit  from  the  conduct  of  a  set  of  controlled  experiments 
and  testing  efforts  that  could  begin  to  isolate  the  specific  attributes  and  associated  available 
measures  that  discriminate  successful  from  unsuccessful  analysts. 

Explore  the  use  of  nonability-based  measures,  such  as  biodata,  for  analyst  selection  and 
assignment. 

A  common  theme  in  the  literature  and  in  our  interviews  with  analyst  SMEs  was  that  the  amount 
and  diversity  of  analyst  experience  is  a  good  predictor  of  the  quality  of  analysis  that  an  analyst 
will  produce  and  the  success  he  or  she  will  likely  have  in  the  field.  As  such,  an  effort  to  describe 
and  categorize  the  types  of  experiences  that  are  of  value  to  the  analyst  could  lead  to  the 
development  of  a  life  history  measure  that  would  identify  the  candidate  analyst  who  could  bring 
to  the  job  a  set  of  experiences  that  could  predispose  him  or  her  to  success  in  the  analysis  domain. 

4.3  Far  Term 

Examine  the  trainability  of  important  abilities  for  successful  IAs. 

Much  of  the  focus  of  the  current  effort  was  on  identifying  key  analyst  attributes  that  could  be 
quantified  through  existing  measures  and  understanding  the  impact  those  attributes  have  on 
analyst  performance.  The  results  of  the  effort  point  to  a  number  of  key  analyst  attributes  that 
contribute  to  effective  performance.  The  ability  to  measure  the  attribute  is  a  necessary  precursor 
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for  both  including  a  measure  of  the  attribute  as  part  of  the  analyst  selection  criteria  and  also  for 
potentially  introducing  training  to  the  analyst  program  of  instruction  aimed  specifically  at 
improving  some  of  these  attributes.  The  Intelligence  School  may  benefit  from  an  effort  to 
understand  which  of  the  critical  attributes  can  be  trained,  so  that  training  materials  or  courses  can 
be  developed  (or  purchased)  to  improve  analyst  proficiency  on  specific  attribute -related 
activities. 

Investigate  research  areas  involving  neural  (EEG)  mapping  that  may  help  link  higher-level 
cognitive  performance  (e.g.,  reasoning,  strategy  use,  decision  making,  etc.)  to  lower-level  IP 
activities  that  support  and  enhance  the  higher-level  cognitive  performance  outcomes. 

The  area  of  neural  (EEG)  mapping  has  made  significant  progress  over  the  past  10  years  and 
provides  the  potential  for  establishing  direct  links  between  overt  performance  in  the  way  of 
strategy  use,  reasoning,  and  decision  making,  and  activities  associated  with  how  humans  store 
and  process  information. 

Perform  a  componential  analysis  of  critical  IA  tasks  to  identify  the  individual  abilities  and 
aptitudes  that  are  correlated  with  (predict)  performance  on  specific  IA  tasks  and  subtasks. 

A  componential  analysis  would  identify  the  specific  independent  aptitudes  and  skills  that  are 
predictive  of  “good”  IA  task  performance,  and  the  weights  (or  importance)  of  each  aptitude  or 
skill  to  overall  performance.  This  would  help  to  develop  a  profile  of  the  “exceptional”  analysts 
and  provide  ways  to  “trade-off”  skills  to  identify  the  potentially  “best”  analyst  given  a  pool  of 
candidates. 
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Appendix  A.  Bibliography  of  Research  Reviewed 


This  appendix  contains  the  full  set  of  literature  we  reviewed  as  part  of  this  project  effort.  The  list 
contains  references  on  the  intelligence  analysis  process,  relevant  performance  metrics  for 
analysis,  the  attributes  required  to  accomplish  the  analysis  process,  and  general  references  to  the 
cognitive  attributes  evaluated  in  this  project.  The  purpose  of  this  appendix  is  to  provide  a 
comprehensive  resource  list  for  the  areas  the  research  team  investigated  and  to  catalogue  those 
resources  that,  although  influential  to  the  project  as  a  whole,  were  not  directly  cited  in  the  final 
report. 
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Appendix  B.  Summary  of  Initial  Literature  Review 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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B.l  Sternberg  Triarchic  Theory/Componential  Analysis  References 

Overall  Summary:  Sternberg’s  theory  of  intelligence  is  focused  on  identifying  and  measuring 
how  individuals  process  information.  His  theory  states  that  an  individual’s  environment  or  world 
context  is  critical  for  identifying  and  understanding  intelligent  behavior  and  that  one’s 
knowledge  about  their  own  (strategic)  thinking  processes  (i.e.,  metacognition)  as  well  as  their 
ability  to  evaluate  and  modify  their  cognitive  strategies  (i.e.,  executive  control  processes)  are  key 
elements  of  intelligence. 

The  theory  posits  a  subtheory  of  human  intelligence  based  on  component  constructs  that  act  at 
varying  levels:  (a)  metacomponents  (higher  order  planning,  monitoring,  evaluating,  etc.),  (b) 
performance  (executing  problem-solving  strategy),  (c)  acquisition  (learning  new  strategy),  (d) 
retention  (retrieving  stored  knowledge),  and  (e)  transfer  (applying  learned  strategy  to  other 
situations).  By  analyzing  these  component  constructs  via  “componential  analysis”  techniques, 
one  can  presumably  isolate  their  impact  on  performance,  as  well  as  develop  training  that  can 
improve  performance. 

Sternberg  has  used  cognitive  constructs  to  decompose  selected  psychometric  tests  as  a  way  to 
better  isolate,  understand  and  measure  the  specific  information-processing  elements  that 
influence  test  performance. 

One  interesting  aspect  to  Sternberg’s  theory  is  that  as  an  individual’s  metacognitive  processes 
become  more  effective  and  efficient  (i.e.,  automated),  then  mental  capacity  may  be  freed  up  for 
other  important  information-processing/strategic  thinking  activities  (e.g.,  situational  awareness, 
identifying  viable  alternative  strategies/options,  creativity,  etc.).  While  this  view  is  based  on  the 
study  team’s  understanding  of  Sternberg’s  theory  and  thus  has  yet  to  be  demonstrated 
empirically,  it  may  help  to  provide  a  future  area  of  research  that  could  lead  to  improved 
information-processing  capability  of  IA  personnel. 

Bettman,  J.R.,  Johnson,  E.J.,  and  Payne,  J.W.  (1990).  A  componential  analysis  of  cognitive 
effort  in  choice.  Organizational  Behavior  and  Human  Decision  Processes,  45  (1), 
111-139. 

Summary:  Describes  use  of  elementary  information  processes  (EIPs)  that  are 
required  to  execute  a  given  decision  strategy.  These  component  processes  form 
the  basis  of  a  model  that  can  be  used  to  predict  mental  effort  needed  to  execute  a 
decision  strategy.  Authors  report  that  models  developed  using  this  approach  were 
good  predictors  of  task-specific  response  times  and  subjective  reporting  of  task- 
related  effort. 

Sternberg,  R.J.  (1985).  Beyond  IQ:  A  triarchic  theory  of  intelligence.  Cambridge 
University  Press,  New  York,  NY. 
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Summary:  Describes  Sternberg’s  Triarchic  Theory  as  having  3  sub-theories: 
Contextual,  experiential  and  componential.  Contextual  sub-theory  relates 
intelligence  to  what  is  going  on  in  the  world  around  the  individual  and  answers 
the  “for  whom  and  where”  behaviors  are  intelligent.  Experiential  sub-theory 
relates  intelligence  to  both  the  external  and  internal  worlds  of  the  individual  and 
answers  the  “when”  behaviors  are  intelligent.  The  Componential  sub-theory 
relates  intelligence  to  the  internal  world  of  the  individual  and  answers  the  “how” 
behaviors  are  intelligent.  This  last  area  describes  the  mental  mechanisms  that 
underlie  intelligent  behavior. 

Sternberg,  R.J.  (1981).  Testing  and  cognitive  psychology.  American  Psychologist ,  36  (10), 
1181-1189. 

Summary:  Describes  four  major  approaches  to  understanding  mental  abilities: 

(1)  Cognitive  correlates,  (2)  cognitive  components,  (3)  cognitive  training,  and 
(4)  cognitive  contents.  Taken  together  these  approaches  help  one  understand 
performance  differences  on  psychometric  tests  due  to  the  underlying  mental 
processes  required  to  perform  a  given  test. 

B.2  Neural  Activity  (EEG)  Tracking  References 

Overall  Summary:  The  ability  to  measure  neurological  functioning  and  relate  it  to  specific 
cognitive  functioning  has  grown  steadily  over  the  past  10-15  years.  Of  direct  importance  to  this 
project  are  studies  that  show  a  relationship  between  specific  neurological  activity  and  short-term 
(working)  memory  as  well  as  other  (strategic)  functioning  that  can  used  to  predict  performance 
on  complex  tasks.  Another  intriguing  finding  (see  Jaeggi  et.  al.,  2008)  is  the  potential  for 
improving  cognitive  processes  related  to  fluid  intelligence  that  can  transfer  to  other,  unrelated 
tasks. 

Gevins,  A.,  Smith,  M.E.,  McEvoy,  L.K.,  and  Yu,  D.  (1997).  High-resolution  EEG  mapping 
of  cortical  activation  related  to  working  memory:  Effects  of  task  difficulty,  type 
processing  and  practice.  Cerebral  Cortex,  7  (4),  374-385 

Summary:  Presents  evidence  of  brain  activity  associated  with  working  memory 
tasks  and  that  as  practice  increased  there  was  a  reduction  in  this  activity.  In 
addition,  some  support  was  found  for  identification  of  a  “coherent  functional 
network”  of  cortical  activity  when  subjects  were  presented  with  more  difficult 
memory  tasks. 

Gevins,  A.  and  Smith,  M.E.  (2000).  Neurophysiological  measures  of  working  memory  and 
individual  differences  in  cognitive  ability  and  cognitive  style.  Cerbral  Cortex,  10  (9), 
829-839. 


51 


Summary:  Describes  between-subject  differences  in  brain  activity  related  to  the 
capacity  to  deliberately  control  attention  in  order  to  manipulate  information  in 
working  memory.  Results  indicate  that  high-ability  subjects  developed  strategies 
that  made  greater  use  of  parietal  brain  regions  versus  low-ability  subjects  more 
exclusive  use  of  frontal  regions.  Results  reflect  ability  to  show  brain-related 
activity  associated  with  differing  cognitive  ability  and  style. 

Jaeggi,  S.M.,  Buschkuehl,  M.  Jonides,  J.,  and  Perrig,  W.J.  (2008).  Improving  fluid 

intelligence  with  training  on  working  memory.  Proceedings  of  the  National  Academy  of 
Sciences  of  the  United  States  of  America  (PNAS),  May  13,  2008  vol.  105 

Summary:  Presents  evidence  of  training  transfer  from  a  demanding  memory  task  to 
measure  of  fluid  intelligence  (Gf)  despite  the  memory  task  being  very  different 
than  Gf  test  activities.  Gf  refers  to  the  ability  to  reason  and  solve  new  problems 
independently  of  previously  acquired  knowledge.  Training  transfer  impact 
thought  to  be  “dosage-dependent”  with  greater  impact  occurring  with  exposure  to 
more  training. 

Smith,  M.E.,  McEvoy,  L.K.,  and  Gevins,  A.  (1999).  Neurophysiological  indices  of  strategy 
development  and  skill  acquisition.  Cognitive  Brian  Research ,  7  (3),  389-404. 

Summary:  Describes  task  specific  practice  effects  related  to  a  verbal  working 
memory  task.  This  finding  was  thought  to  provide  neurophysiological  evidence 
for  the  formation  of  a  task-specific  neurocognitive  strategy.  In  addition,  these 
results  suggest  the  EEG  can  be  used  to  monitor  practice-related  changes  to  brain 
activities  associated  with  specific  task  processing  as  well  as  development  of 
strategies  that  impact  performance. 

B.3  Cognitive  Load  Theory  (CLT)  References 

Overall  Summary:  Much  of  the  CLT  literature  is  focused  on  the  learning  and  instruction, 
however,  it  may  provide  an  overall  approach  and  measures  for  determining  how  good/poor 
intelligence  analysts  are  processing  information,  or  more  precisely,  the  cognitive  effort  required 
to  process/analyze  the  information  and  make  associate  decisions. 

One  measurement  technique  used  by  CLT  researchers  involves  the  use  of  (simultaneous)  dual¬ 
tasks.  That  is,  a  secondary  task  is  used  to  assess  the  effort  required  to  complete  the  primary  task. 
Since  it  is  done  simultaneously,  it  can  provide  a  real-time  measure  of  mental  effort  being 
expended. 

It  seems  that  more  proficient  IAs  will  have  available  to  them  greater  cognitive  capacity  when 
completing  selected  job  tasks  and  would  thus  be  able  to  complete  tasks  quicker  and  with  greater 
accuracy.  Part  of  the  reason  for  this  is  good  analysts  might  have  available  to  them  one  or  more 
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useful  schemas  (e.g.,  for  identifying  useful  info,  for  developing  appropriate  problem-solving 
strategies,  etc.). 

Individual  Article  Summaries: 

Arino,  A.R.  (2008).  Cognitive  load  theory  and  the  role  of  learner  experience:  An 
abbreviated  review  for  educational  practitioners.  AACE  Journal ,  16  (4),  425-439. 

Summary:  Describes  three  types  of  cognitive  load  sources:  Intrinsic,  extraneous, 
and  germane.  Intrinsic  load  refers  to  the  number  of  elements  that  must  be 
processed  simultaneously  in  working  memory.  Extraneous  load  is  considered 
“ineffective”  due  to  it  requiring  person  to  engage  working  memory  that  is  not 
directly  related  to  a  schema.  Germane  load  is  considered  “effective”  due  to 
person  able  to  process  information  in  manner  that  is  relevant  to  a  given  schema. 

Ippel,  M.J.  (1996).  Cognitive  task  load  and  test  performance.  Paper  presented  at  the 

International  Military  Testing  Association  (IMTA),  San  Antonio,  TX,  Nov  12-14,  1996. 

Summary:  Presents  elements  of  a  theory  of  measurement  of  the  effects  of 
information  overload  on  task  performance  as  measured  by  accuracy  scores.  A 
promising  methodology,  called  the  “facet  design  technique,”  is  also  described  that 
may  lead  to  identifying  performance  profiles  based  on  task  facets. 

Gwizdka,  J.  (2010).  Using  Stroop  task  to  assess  cognitive  load.  Proceedings  of  the  2010 
European  Conference  on  Cognitive  Ergonomics.  Delft,  Netherlands,  Aug  25-27,  2010. 

Summary:  Describes  use  of  a  Stroop-like  task  as  the  secondary  task  within  a  dual¬ 
task  paradigm  with  the  results  indicating  the  ability  to  reliably  identifying 
extraneous  versus  intrinsic  task  load. 

Paas,  F.,  Tuovinen,  J.E.,  Tabbers,  H.,  and  Van  Gerven,  P.W.  (2003).  Cognitive  load 

measurement  as  a  means  to  advance  cognitive  load  theory.  Educational  Psychologist,  38 
(1),  63-71. 

Summary:  Describes  CLT  and  learning  transfer:  That  is,  transfer  of  existing 
knowledge/experience  to  new  situations.  According  to  the  authors,  the  results 
show  how  a  combination  of  performance  and  cognitive  load  measures  are  a 
reliable  indicator  of  mental  efficiency  of  instructional  methods. 

Sweller,  J.  (1988)  Cognitive  load  during  problem  solving:  Effects  on  learning.  Cognitive 
Science,  12,  257-285. 

Summary:  Presents  research  related  to  use  of  means-ends  analysis  and  schema 
acquisition  (tasks)  within  a  dual-task  paradigm  to  more  precisely  measure 
cognitive  load  within  a  problem  solving  search  situation. 
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Intentionally  lelt  blank. 
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Appendix  C.  Standardized  Test  Summary 


This  appendix  appears  in  its  original  form,  without  editorial  change. 
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Table  C-l  contains  descriptions  of  standardized  tests  that  target  one  or  more  of  the  cognitive  abilities. 

Table  C-l.  Summary  of  standardized  cognitive  ability  tests. 


Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

"Business 
Critical 
Thinking 
Skills  Test" 

deductive 

reasoning, 

inductive 

reasoning 

Analysis,  Inference,  Evaluation,  Deductive,  Inductive,  Total. 

'Developed  to  assess  the  critical  thinking 
skills  of  business  professionals  and 
business  students.' 

"Aptitude 

Interest 

Inventory" 

inductive 

reasoning 

Aptitude  Based  Career  Decision  Test:  7  scores:  Clerical 

Perception,  Vocabulary,  Numerical  Computation,  Numerical 
Reasoning,  Spatial  Visualization,  Inductive  Reasoning,  Analytical 
Reasoning;  Interest  Based  Career  Decision  Test:  ratings  in  3  areas: 
Data,  People,  Things. 

To  develop  a  plan  for  career  exploration. 

"Learning 

Ability 

Profile" 

deductive 

reasoning, 

inductive 

reasoning 

Total  and  4  derived  scores  (Certainty,  Problem  Solving,  Flexibility, 
Frustration). 

Designed  to  provide  'a  quantifiable  index 
of  the  subject's  inductive  and  deductive 
reasoning,  cognitive  and  problem  solving 
skills'. 

"Test  of 

Everyday 

Reasoning" 

deductive 

reasoning, 

inductive 

reasoning 

Analysis,  Evaluation,  Inference,  Deductive  Reasoning,  Inductive 
Reasoning,  Total. 

Designed  to  supplement  information  on 
applications  for  employment,  educational 
assessments,  and  program  evaluations  by 
assessing  basic  reasoning  skills. 

"Health 

Sciences 

Reasoning 

Test" 

deductive 

reasoning, 

inductive 

reasoning 

Analysis,  Evaluation,  Inference,  Deductive  Reasoning,  Inductive 
Reasoning,  Total. 

'Was  developed  for  use  by  educators  and 
researchers  to  assess  the  critical  thinking 
skills  of  health  science  professionals  and 
health  science  students'. 

Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

"Cornell 

Critical 

Thinking 

Tests" 

deductive 

reasoning, 

inductive 

reasoning 

Total  score  only  for  each  level. 

Assesses  general  critical  thinking  ability 
including  'induction,  deduction,  evaluation, 
observation,  credibility  (of  statements 
made  by  others),  assumption  identification, 
and  meaning'. 

"deductive 

reasoning 

test" 

deductive 

reasoning 

Total  score  only. 

'Intended  as  a  selection  instrument  for 
scientific,  entrepreneurial  and  other  high 
level  professional  and  occupational 
personnel'. 

"Ball 

Aptitude 

Battery" 

inductive 

reasoning 

Clerical,  Analytical  Reasoning,  Inductive  Reasoning,  Vocabulary, 
Numerical  Computation,  Numerical  Reasoning,  Paper  Folding, 
Writing  Speed,  Associative  Memory,  Auditory  Memory  Span,  Idea 
Generation,  Word  Association. 

Designed  to  measure  'various  aptitudes 
needed  for  successful  performance  in  a 
wide  variety  of  educational  and  work 
settings.' 

"Comprehen 
sive  Ability 
Battery" 

inductive 

reasoning, 

Flexibility  of 

Closure, 

Originality/Cre 

ativity 

Verbal  Ability  (V),  Numerical  Ability  (N),  Spatial  Ability  (S), 

Speed  of  Closure  (Cs),  Perceptual  Speed  and  Accuracy  (P), 

Inductive  Reasoning  (I),  Flexibility  of  Closure  (Cf),  Associative 
Memory  (Ma),  Mechanical  Ability  (Mk),  Memory  Span  (Ms), 
Meaningful  Memory  (Mm),  Spelling  (Sp),  Auditory  Ability  (AA), 
Esthetic  Judgment  (E),  Spontaneous  Flexibility  (Fs),  Ideational 
Fluency  (Fi),  Word  Fluency  (W),  Originality  (0),  Aiming  (A), 
Representational  Drawing  (RD). 

'Features  20  tests,  each  designed  to 
measure  a  single  primary  ability  factor  .  .  . 
important  in  industrial  settings  and  career 
and  vocational  counseling.' 

"California 
Critical 
Thinking 
Skills  Test" 

deductive 

reasoning, 

inductive 

reasoning 

Analysis,  Inference,  Evaluation,  Deductive  Reasoning,  Inductive 
Reasoning,  Total  Score. 

'Specifically  designed  to  measure  the  skills 
dimension  of  critical  thinking.' 

"Watson- 

Glaser 

Critical 

Thinking 

Appraisal" 

deductive 

reasoning 

Inference,  Recognition  of  Assumptions,  Deduction,  Interpretation, 
Evaluation  of  Arguments,  Total. 

Constructed  to  assess  critical  thinking 
abilities  related  to  reading  comprehension. 

Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

"Assessment 

Inventory 

for 

Management 

M 

deductive 

reasoning, 

inductive 

reasoning 

Staffing/Recruiting  and  Selection,  Training,  Performance 
Management— Supervision,  Business  Management,  Lield  Office 
Development,  Interpersonal  Relations  (Communicating, 

Counseling,  Supporting),  Leadership  (Delegating,  Motivating, 
Rewarding,  Team  Building,  Networking),  Organization 
(Coordinating,  Monitoring,  Planning,  Problem-Solving  and 
Decision-Making) . 

Lor  screening  candidates  for  insurance 
field  management  positions.' 

"Jouve- 

Cerebrals 

Test  of 
Induction" 

inductive 

reasoning 

It  yields  with  a  Reasoning  Index  (RIX),  an  age -referenced  standard 
score  that  uses  a  mean  of  100,  and  15  points  per  standard 
deviation. 

The  JCTI  is  a  computerized  nonverbal 
intelligence  test  designed  with  figurative 
items  which  don’t  require  acquired 
knowledge.  The  JCTI  has  been  developed 
so  that  the  number  of  items  (up  to  52)  is 
individually  adapted  according  to  the  test- 
taker’s  performance.  Thanks  to  the 
nonverbal  nature  of  its  items,  this  test  is 
suitable  for  testing  persons  without  cultural 
bias. 

"Emotional 

Competence 

Inventory" 

Collaboration 

Self-Awareness  (Emotional  Self-Awareness,  Accurate  Self- 
Assessment,  Self-Confidence);  Self-Management  (Emotional  Self- 
Control,  Transparency,  Adaptability,  Achievement  Orientation, 
Initiative,  Optimism);  Social  Awareness  (Empathy,  Organizational 
Awareness,  Service  Orientation);  Relationship  Management 
(Developing  Others,  Inspirational  Leadership,  Influence,  Change 
Catalyst,  Conflict  Management,  Teamwork  &  Collaboration). 

'Designed  to  assess  emotional  intelligence 
(the  ability  to  recognize  and  manage 
emotions  [yours  and  other]).' 

"Lore 

Leadership 

Assessment" 

Collaboration 

Overall  Leadership  Effectiveness,  Moral  Leadership,  Intellectual 
Leadership,  Courage,  Collaboration,  Visionary/Inspirational. 

Designed  to  'measure  behaviors,  skills,  and 
impacts  desirable  in  a  leader.' 

"Change 

Abilitator" 

Collaboration 

Information,  Personal,  Operational,  Impact,  Collaboration, 
Transforming. 

Designed  to  identify  six  types  of  concern 
people  experience  when  change  is 
introduced  into  their  organization. 

Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

"Assessment 

of  Individual 

Learning 

Style: 

Perceptual 

Memory 

Task" 

"memory"  - 

Recall, 

Recognition 

7  scores,  3  alternate  scores:  Spatial  Relations,  Visual  Designs 
Recognition,  Visual  Designs-Sequencing,  Auditory-Visual  Colors 
Recognition,  Auditory -Visual  Colors  Sequencing,  Discrimination 
Recall,  Total  PMT,  Visual-Visual  (alternate),  Auditory-Auditory 
(alternate),  Visual-Auditory  (alternate). 

'To  provide  measures  of  the  individual's 
perception  and  memory  for  spatial 
relationships;  visual  and  auditory 
sequential  memory;  intermediate  term 
memory;  and  discrimination  of  detail'. 

"Bloomer 

Learning 

Test" 

"memory"  - 
Recall 

Activity,  Response  Integration,  Boredom,  Visual  Short-Term 
Memory,  Auditory  Short-Term  Memory,  Visual  Apprehension 

Span,  Impulse,  Stimulus  Complexity,  Serial  Learning,  Recall, 
Relearning,  Learning  Set,  Free  Association,  Emotional  Ratio, 

Paired  Associate  Learning,  Paired  Associate  Decrement, 
Interference,  Concept  Recognition,  Concept  Production,  Problem 
Solving,  yielding  3  IQ  scores  (Simple  Learning,  Problem  Solving, 
Full  Learning) 

Identifies  'academic  difficulties  on  the 
basis  of  a  pattern  of  strengths  and 
weaknesses  in  various  learning  processes 

"Kaplan- 

Baycrest 

Neuropsych 

ological 

Assessment" 

"memory"  - 

Recall, 

Recognition 

Attention/Concentration,  Immediate  Memory  Recall,  Delayed 
Memory  Recall,  Delayed  Memory  Recognition,  Spatial  Processing, 
Verbal  Fluency,  Reasoning/Conceptual  Shifting,  Total  Index. 

'Test  of  neurocognitive  functioning.' 

"Doors  and 
People:  A 

Test  of 

Visual  and 
Verbal 

Recall  and 
Recognition 

M 

"memory"  - 
Verbal 

Memory 

Verbal  Recall  (People),  Visual  Recognition  (Doors),  Visual  Recall 
(Shapes),  Verbal  Recognition  (Names),  Overall,  Combined  Visual 
Memory,  Combined  Verbal  Memory,  Combined  Recall,  Combined 
Recognition,  Forgetting  (Verbal),  Forgetting  (Visual),  Overall 
Forgetting,  Visual-Verbal  Discrepancies,  Recall-Recognition 
Discrepancies. 

Designed  to  'provide  comparable  measures 
of  visual  and  verbal  memory'  and  'test  both 
recall  and  recognition.' 

Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

"Kaufman 

Short 

Neuropsych 

ological 

Assessment 

Procedure" 

"memory"  - 
Recall 

Gestalt  Closure,  Number  Recall,  Four-Letter  Words, 

Recall/Closure  Composite,  K-SNAP  Composite. 

Constructed  to  assess  the  'ability  to 
demonstrate  intact  mental  functioning.' 

"Learning 

Efficiency 

Test" 

"memory"  - 
Recall,  Visual 
Memory 

Visual  Ordered  Recall  (Immediate  Recall,  Short  Term  Recall, 

Long  Term  Recall)  Visual  Unordered  Recall  (Immediate  Recall, 
Short  Term  Recall,  Long  Term  Recall),  Auditory  Ordered  Recall 
(Immediate  Recall,  Short  Term  Recall,  Long  Term  Recall), 

Auditory  Unordered  Recall  (Immediate  Recall,  Short  Term  Recall, 
Long  Term  Recall),  Total  Visual  Memory,  Total  Auditory 

Memory,  Global  Memory. 

'Yields  information  about  a  person's 
preferred  modality  for  learning  and 
provides  insights  about  the  impact  of 
interference  on  memory  storage  and 
retrieval,  and  the  kinds  of  metacognitive 
strategies  used  during  learning'. 

"Swanson- 

Cognitive 

Processing 

Test" 

"memory"  - 
Recall 

Rhyming  Words,  Visual  Matrix,  Auditory  Digit  Sequence, 

Mapping  and  Directions,  Story  Retelling,  Picture  Sequence,  Phrase 
Recall,  Spatial  Organization,  Semantic  Association,  Semantic 
Categorization,  Nonverbal  Sequence,  Semantic,  Episodic,  Total, 
Auditory,  Visual,  Prospective,  Retrospective,  Strategy  Efficiency 
Index,  Processing  Difference  Index,  Instructional  Efficiency  Index, 
Stability  Index. 

Designed  to  assess  'different  aspects  of 
mental  processing  ability  and  potential.' 

"Schaie- 

Thurstone 

Adult 

Mental 

Abilities 

Test" 

"memory"  - 
Recognition 

Recognition  Vocabulary,  Figure  Rotation,  Letter  Series,  Number 
Addition,  Word  Fluency,  Object  Rotation,  Word  Series. 

'Measuring  the  mental  abilities  of  adults'. 

"Non-Verbal 

Ability 

Tests" 

"memory"  - 
Recall 

Matching  Shape,  Matching  Direction,  Categorization,  Picture 
Completion,  Embedded  Figures,  Figure  Formation,  Mazes, 
Sequencing,  Picture  Arrangement,  Visual  Search,  Simple  Key 

Test,  Complex  Key  Test,  Code  Tracking  I,  Code  Tracking  II, 

Visual  Recognition,  Auditory  Recognition,  Auditory  Recall,  Visual 
Recall,  Total. 

Designed  to  measure  perceptual, 
conceptual,  attention/concentration,  and 
memory  skills. 

Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

"Structure  of 
Intellect 
Learning 
Abilities 

Test" 

"memory"  - 
Visual 

Memory 

26  subtests  in  5  test  areas  Cognition  (Cognition  of  Figural  Units, 
Cognition  of  Figural  Classes,  Cognition  of  Figural  Systems, 
Cognition  of  Figural  Transformations,  Cognition  of  Symbolic 
Relations,  Cognition  of  Symbolic  Systems,  Cognition  of  Semantic 
Units,  Cognition  of  Semantic  Relations,  Cognition  of  Semantic 
Systems),  Memory  (Memory  of  Figural  Units,  Memory  of 

Symbolic  Units— Visual,  Memory  of  Symbolic  Systems— Visual, 
Memory  of  Symbolic  Units— Auditory,  Memory  of  Symbolic 
Systems— Auditory,  Memory  of  Symbolic  Implications), 

Evaluation  (Evaluation  of  Figural  Units,  Evaluation  of  Figural 
Classes,  Evaluation  of  Symbolic  Classes,  Evaluation  of  Symbolic 
Systems),  Convergent  Production  (Convergent  Production  of 

Figural  Units,  Convergent  Production  of  Symbolic  Systems, 
Convergent  Production  of  Symbolic  Transformations,  Convergent 
Production  of  Symbolic  Implications),  and  Divergent  Production 
(Divergent  Production  of  Figural  Units,  Divergent  Production  of 
Semantic  Units,  and  Divergent  Production  of  Symbolic  Relations) 
yielding  14  general  ability  scores  Cognition,  Memory,  Evaluation, 
Convergent  Production,  Divergent  Production,  Figural,  Symbolic, 
Semantic,  Units,  Classes,  Relations,  Systems,  Transformations, 
and  Implications. 

'Designed  to  assess  a  wide  variety  of 
cognitive  abilities  or  factors  of  intelligence 
in  children  and  adults'. 

"Concealed 

Figures" 

Flexibility  of 
Closure 

Total  score 

Assesses  'the  ability  to  hold  a  configuration 
in  mind  despite  distraction.' 

"Visual 

Patterns 

Test" 

"memory"  - 
Visual 

Memory 

Total  score  only. 

Designed  to  measure  'short  term  visual 
memory.' 

"Randt 

Memory 

Test" 

"memory"  - 
Recall 

Acquisition  Recall  (Five  Items  Acquisition,  Paired  Words 
Acquisition,  Short  Story  Verbatim,  Digit  Span,  Incidental 

Learning),  Delayed  Recall  (Five  Items  Recall,  Paired  Words 

Recall,  Picture  Recall,  General  Information),  and  Memory  Index. 

Functions  'as  a  global  survey  and 
evaluation  of  patients'  complaints 
concerning  their  memory'. 

Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

"Personal 

Characteristi 

cs 

Inventory" 

"memory"  - 
Recognition 

Agreeableness  (Cooperation,  Consideration,  Total),  Extraversion 
(Sociability,  Need  for  Recognition,  Leadership  Orientation,  Total), 
Conscientiousness  (Dependability,  Achievement  Striving, 

Efficiency,  Total),  Stability  (Even  Temperament,  Self-Confidence, 
Total),  Openness  (Abstract  Thinking,  Creative  Thinking,  Total), 
Occupational  Score  (Manager,  Sales,  Clerical,  Production,  Driver), 
Teamwork,  Integrity,  Learning  Orientation,  Commitment  to  Work. 

Designed  to  'help  organizations  hire  more 
effectively'  and  provide  feedback  to 
employees  regarding  'strengths  and  areas 
where  improvement  is  necessary';  used  in 
the  hiring  process  and  for  developmental 
purposes. 

"Reynolds 

Intellectual 

Assessment 

Scales" 

"memory"  - 
Verbal 

Memory 

Verbal  Intelligence  Index  (Guess  What,  Verbal  Reasoning), 
Nonverbal  Intelligence  Index  (Odd-Item  Out,  What's  Missing), 
Composite  Intelligence  Index,  Composite  Memory  Index  (Verbal 
Memory,  Nonverbal  Memory),  RIST  Index  (Guess  What,  Odd- 
Item  Out). 

Designed  to  assess  verbal  and  nonverbal 
intelligence  and  memory. 

"Continuous 

Visual 

Memory 

Test" 

"memory"  - 

Recognition, 

Visual 

Memory 

Acquisition  (Hits,  False  Alarms,  d-Prime,  Total),  Delayed 
Recognition,  Visual  Discrimination 

Constructed  to  assess  recognition  memory, 
perception,  and  discrimination. 

"Memory 

Assessment 

Scales" 

"memory"  - 
Verbal 

Memory, 

Visual 

Memory 

Short-Term  Memory  (Verbal  Span,  Visual  Span,  Total),  List 
Acquisition,  Delayed  List  Recall,  Delayed  Prose  Recall,  Global 
Memory  Scale  (Verbal  Memory  [List  Recall,  Immediate  Prose 
Recall,  Total],  Visual  Memory  [Visual  Reproduction,  Immediate 
Visual  Recognition,  Total],  Total),  Delayed  Visual  Recognition, 
Names-Faces  (Immediate,  Delayed)  and  7  Verbal  Process  scores: 
Total  Intrusions,  List  Clustering  (Acquisition,  Recall,  Delayed 
Recall),  Cued  List  Recall  (Recall,  Delayed  Recall),  List 

Recognition. 

Developed  to  assess  areas  of  cognitive 
function  that  are  involved  in  memory. 

Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

"Test  of 
Memory  and 
Learning" 

"memory"  - 

Recall, 

Recognition, 

Visual 

memory, 

Verbal 

memory, 

Concentration 

10  Verbal  subtest  scores  (Memory  for  Stories,  Word  Selective 
Reminding,  Object  Recall,  Paired  Recall,  Digits  Forward,  Letters 
Forward,  Digits  Backward,  Letters  Backward,  Memory  for  Stories 
Delayed,  Word  Selective  Reminding  Delayed);  6  Nonverbal 
subtest  scores  (Facial  Memory,  Abstract  Visual  Memory,  Visual 
Sequence  Memory,  Memory  for  Location,  Visual  Selective 
Reminding,  Manual  Imitation);  3  core  composite  scores  (Verbal 
Memory  Index,  Nonverbal  Memory  Index,  Composite  Memory 
Index);  6  supplemental  composite  scores  (Verbal  Delayed  Recall 
Index,  Attention/Concentration  Index,  Sequential  Recall  Index, 

Free  Recall  Index,  Associative  Recall  Index,  Learning  Index). 

Designed  to  assess  the  'key  features  of 
memory'  and  to  'evaluate  learning  as 
reflected  in  changes  in  recall  and 
recognition  over  multiple  trials  of  various 
stimuli.' 

"Brief 

Visuospatial 

Memory 

Test- 

Revised" 

"memory"  - 
Recall, 
Recognition, 
Visual 

Memory 

Total  Recall,  Learning,  Delayed  Recall,  Percent  Retained, 
Recognition  Hits,  Recognition  False  Alarms,  Recognition 
Discrimination  Index,  Recognition  Response  Bias,  Copy 
(Optional) 

Designed  as  an  equivalent,  multiple -test 
form  assessment  of  visual  memory. 

"Rey 

Complex 
Figure  Test 
and 

Recognition 

Trial" 

"memory"  - 
Recall 

Immediate  Recall,  Delayed  Recall,  Recognition  Total  Correct, 

Copy,  Time  to  Copy,  Recognition  True  Positives,  Recognition 

False  Positives,  Recognition  True  Negatives,  Recognition  False 
Negatives. 

Designed  to  'investigate  visuospatial 
constructional  ability  and  visual  memory.' 

"Leiter 

International 

Performance 

Scale- 

Revised" 

"memory"  - 
Recognition 

Visualization  and  Reasoning  (Figure  Ground,  Design  Analogies, 
Form  Completion,  Matching,  Sequential  Order,  Repeated  Patterns, 
Picture  Context,  Classification,  Paper  Folding,  Figure  Rotation), 

VR  Composite  (Fluid  Reasoning,  Brief  IQ,  Fundamental 
Visualization,  Spatial  Visualization,  Full  IQ),  Attention  and 

Memory  Associated  Pairs,  Immediate  Recognition,  Forward 
Memory,  Attention  Sustained,  Reverse  Memory,  Visual  Coding, 
Spatial  Memory,  Delayed  Pairs,  Delayed  Recognition,  Attention 
Divided),  AM  Composite  (Memory  Screen,  Associative  Memory, 
Memory  Span,  Attention,  Memory  Process,  Recognition  Memory). 

Constructed  as  a  'nonverbal  cognitive 
assessment.' 

Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

"Wide 

Range 
Assessment 
of  Memory 
and 

Learning" 

"memory"  - 
Recall, 
Recognition, 
Verbal 

Memory, 

Visual 

Memory 

Verbal  Memory  (Story  Memory,  Verbal  Learning,  Total),  Visual 
Memory  (Design  Memory,  Picture  Memory,  Total), 
Attention/Concentration  (Finger  Windows,  Number  Letter,  Total), 
General  Memory;  Optional  scores:  Working  Memory  (Verbal 
Working  Memory,  Symbolic  Working  Memory,  Total),  Verbal 
Recognition  (Story  Recognition,  Verbal  Learning  Recognition, 
Total),  Visual  Recognition  (Design  Recognition,  Picture  Memory 
Recognition,  Total),  General  Recognition,  Sound  Symbol, 

Sentence  Memory,  Story  Memory  Recall,  Verbal  Learning  Recall, 
Sound  Symbol  Recall. 

Designed  for  use  in  'clinical  assessments  of 
memory  including  evaluation  of  immediate 
and/or  delay  recall  as  well  as 
differentiating  between  verbal,  visual  or 
more  global  memory  deficits.' 

"Repeatable 
Battery  for 
the 

Assessment 

of 

Neuropsych 

ological 

Status" 

"memory"  - 

Recall, 

Recognition 

Immediate  Memory  (List  Learning,  Short  Memory,  Total), 
Visuospatial/Constructional  (Figure  Copy,  Line  Orientation, 

Total),  Language  (Picture  Naming,  Semantic  Fluency,  Total), 
Attention  (Digit  Span,  Coding,  Total),  Delayed  Memory  (List 

Recall,  List  Recognition,  Story  Memory,  Figure  Recall,  Total), 

Total. 

Designed  to  measure  'attention,  language, 
visuospatial/constructional  abilities,  and 
immediate  and  delayed  memory.' 

"Delis- 

Kaplan 

Executive 

Function 

System" 

"memory"  - 
Recognition 

Visual  Scanning,  Number  Sequencing,  Letter  Sequencing, 
Number-Letter  Switching,  Motor  Speed,  Composite  Score,  Letter 
Fluency,  Category  Fluency,  Category  Switching,  Filled  Dots, 

Empty  Dots  Only,  Switching,  Color  Naming,  Word  Reading, 
Inhibition,  Inhibition/Switching,  Free  Sorting,  Sort  Recognition, 
Total  Achievement  Score,  Initial  Abstraction  Score,  Total 
Consecutively  Correct 

To  'comprehensively  assess  ...  the  key 
components  of  executive  functions 
believed  to  be  mediated  primarily  by  the 
frontal  lobe 

"Benton 

Visual 

Retention 

Test" 

"memory"  - 
Visual 

Memory 

Omissions,  Distortions,  Perseverations,  Rotations,  Misplacements, 
Size  Errors,  Total  Left,  Total  Right,  Total 

To  assess  visual  perception,  visual 
memory,  and  visuoconstructive  abilities 

Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

"California 

Verbal 

Learning 

Test" 

"memory"  - 

Recall, 

Recognition 

Immediate  Recall  (Trial  1,  Trials  2-5,  Trials  1-5  Total),  Learning 
Slope,  Semantic  Clustering,  Serial  Clustering,  Subjective 

Clustering,  Primacy/Recency  Recall,  Percentage  of  Recall 
Consistency,  List  B  Trial,  Proactive  Interference,  Short-Delay  Free 
Recall,  Retroactive  Interference,  Short-Delay  Cued  Recall,  Long- 
Delay  Free  Recall,  Long-Delay  Free  Recall  Retention,  Long-Delay 
Cued  Recall,  Repetition  Errors,  Synonym/Subordinate  Intrusions, 
Across-List  Intrusions,  Categorical  Intrusions,  Non-Categorical 
Intrusions,  Yes/No  Recognition  Testing,  False-Positive  Errors, 

Total  Recognition  Discriminability,  Source  Recognition 
Discriminability,  Semantic  Recognition  Discriminability,  Novel 
Recognition  Discriminability,  Response  Bias,  Critical  Item 

Analysis,  Forced-Choice  Recognition  [optional]. 

To  'obtain  a  detailed  and  comprehensive 
assessment  of  verbal  learning  and 
memory.' 

"Wechsler 

Memory 

Scale" 

"memory"  - 
Recall,  Verbal 
Memory, 

Visual 

Memory 

6  primary  (Logical  Memory  I,  Logical  Memory  11,  Verbal  Paired 
Associates  I,  Verbal  Paired  Associates  II,  Designs  I,  Designs  II 
[Adult  battery  only],  Visual  Reproduction  I,  Visual  Reproduction 

II,  Spatial  Addition  [Adult  battery  only],  Symbol  Span;  1  optional 
(Brief  Cognitive  Status  Exam);  Five  indices  (Auditory  Memory, 
Visual  Memory,  Visual  Working  Memory  [Adult  battery  only], 
Immediate  Memory,  Delayed  Memory. 

Developed  to  'assess  various  memory  and 
working  memory  abilities'  among 
'individuals  with  suspected  memory 
deficits  or  diagnosed  with  a  range  of 
neurological,  psychiatric,  and 
developmental  disorders.' 

"Mini- 
Mental  State 
Examination 

It 

"memory"  - 
Recall 

Total  score  only  with  items  in  1 1  sections:  Orientation  to  Time, 
Orientation  to  Place,  Registration,  Attention  and  Calculation, 

Recall,  Naming,  Repetition,  Comprehension,  Reading,  Writing, 
Drawing. 

Designed  to  'measure  cognitive  status  in 
adults.’ 

"Hopkins 

Verbal 

Learning 

Text- 

Revised" 

"memory"  - 

Recall, 

Recognition 

Total  Recall,  Delayed  Recall,  Retention,  Recognition 

Discrimination  Index 

A  'brief  assessment  of  verbal  learning  and 
memory  (immediate  recall,  delayed  recall, 
delayed  recognition).' 

"CNS  Vital 
Signs 
Screening 
Battery" 

"memory"  - 
Verbal 

Memory, 

Visual 

Memory 

Memory,  Mental  Speed,  Reaction  Time  Attention,  Cognitive 
Flexibility  for  7  subtests:  Verbal  Memory,  Visual  Memory,  Finger 
Tapping,  Symbol  Digit  Coding,  Stroop,  Shifting  Attention, 
continuous  Performance. 

Designed  to  'assess  neurocognitive  state.' 

Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

"profile  of 

creative 

abilities" 

Originality/Cre 

ativity, 

Problem 

Sensitivity 

Creativity  Index,  Drawing  (New  Elements,  Originality, 

Orientation,  Perspectives,  Total),  Categories  (Fluency,  Flexibility, 
Total),  Home  Rating  Scale  Total,  School  Rating  Scale  Total. 

'Designed  to  measure  the  creative  abilities 
of  students.' 

"Meyer- 

Kendall 

Assessment 

Survey" 

perseverance 

Objectivity,  Social  Desirability  Bias,  Dominance,  Extraversion, 
People  Concerns,  Attention  to  Detail,  Anxiety,  Stability, 
Psychosomatic  Tendencies,  Determination,  Achievement 

Motivation,  Independence. 

Constructed  to  assess  work-related 
personality  style. 

"Work 

Personality 

Index" 

perseverance 

Teamwork,  Concern  for  Others,  Outgoing,  Democratic,  Attention 
to  Detail,  Rule -Following,  Dependability,  Ambition,  Energy, 
Persistence,  Feadership,  Innovation,  Analytic  Thinking,  Self- 
Control,  Stress  Tolerance,  Initiative,  Flexibility,  Achievement, 
Conscientiousness,  Social  Orientation,  Practical  Intelligence, 
Adjustment. 

'Designed  to  identify  personality  traits  that 
directly  relate  to  work  performance.' 

"Inventory 

for 

Counseling 

& 

Developmen 

t" 

perseverance, 

Originality/Cre 

ativity 

Agreement,  Favorable  Impression,  Infrequent,  Insecurity, 

Alienation,  Exam  Tension,  Ambition,  Persistence,  Practicality, 
Sociability,  Teacher-Student  Interaction,  Intellectuality, 

Originality,  Adaptability,  Orderliness,  Fiberal -Conservative, 
Socio-Political  Interest,  Sexual  Beliefs,  Sex  Role  Differences, 
Academic  Performance,  Academic  Excellence,  Academic 

Capacity,  Academic  Motivation. 

An  attempt  to  identify  'strengths,  assets  and 
coping  skills  of  college  students  seeking 
assistance  with  vocational,  educational  and 
personal  problems'. 

"Poppleton 
Allen  Sales 
Aptitude 

Test" 

perseverance 

Administrative  Effectiveness,  Social  Sophistication,  Emotional 
Resilience,  Dynamism,  Economic  Motivation,  Empathy, 
Competitiveness,  Organizational  Ability,  Work  Commitment, 
Emotional  Stability,  Self-sufficiency,  Verbal  Fluency, 

Determination,  Self-confidence,  Entertaining 

'Designed  to  measure  those  attributes 
which  are  of  importance  for  effective 
selling.' 

"Occupation 

al 

Personality 

Assessment" 

perseverance 

Interest  Profile,  Experience  Profile,  How  Much  Interest  and 
Experience  Match,  Focus  on  Making  Money,  Tactical  Orientation, 
Reading  Business  Tables,  Commercial  Orientation,  Personal 
Motives,  Peaceful,  Sensory,  Acquisitive,  Work  Driven,  Work 
Motives,  Intellect,  Beauty,  Independent  Authority,  Pleasing  Others, 
Aggressiveness,  Field  Independence,  Toughness,  Determination, 
Fluctuating  Salary,  Fluctuating  Income  Tolerance,  Management 
Style  Preference. 

Designed  as  a  'computer  simulation  of  a 
vocational  assessment.' 

Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

"Howarth 

Personality 

Questionnair 

e" 

perseverance 

Sociability,  Anxiety,  Dominance,  Conscience,  Hypochondriac- 
Medical,  Impulsive,  Cooperative-Considerateness,  Inferiority, 
Persistence,  Suspicion  vs.  Trust. 

To  measure  personality  dimensions. 

"Athletic 

Motivation 

Inventory" 

perseverance 

Drive,  Aggressiveness,  Determination,  Responsibility,  Leadership, 
Self-Confidence,  Emotional  Control,  Mental  Toughness, 
Coachability,  Conscientiousness,  Trust,  Validity  Scales  (Accuracy, 
Desirability,  Completion  Rate). 

Constructed  to  measure  the  personality  and 
motivation  of  athletes  participating  in 
competitive  sports. 

"Basic 

Attributes 

Test" 

Timesharing 

5  tests  including  Two  Hand  Coordination  (psychomotor),  Complex 
Coordination  (psychomotor),  Item  Recognition  (short  term 
memory),  Time  Sharing  (psychomotor),  and  Activities  Interest 
Inventory  (attitudes) 

A  computerized  test  battery  used  by  the  US 
Air  Force  for  pilot  selection 

"Achieveme 

nt 

Motivation 

Inventory" 

perseverance 

Compensatory  Effort,  Competitiveness,  Confidence  in  Success, 
Dominance,  Eagerness  to  Learn,  Engagement,  Fearlessness, 
Flexibility,  Flow,  Goal  Setting,  Independence,  Internality, 
Persistence,  Preference  for  Difficult  Tasks,  Pride  in  Productivity, 
Self-Control,  Status  Orientation. 

Designed  to  evaluate  'all  major  aspects  of 
job-related  achievement  motivation.' 

"Work 

Personality 

Profile" 

perseverance 

Acceptance  of  Work  Role,  Ability  to  Profit  from  Instruction  or 
Correction,  Work  Persistence,  Work  Tolerance,  Amount  of 
Supervision  Required,  Extent  Trainee  Seeks  Assistance  from 
Supervisor,  Degree  of  Comfort  or  Anxiety  with  Supervisor, 
Appropriateness  of  Personal  Relations  with  Supervisor, 

Teamwork,  Ability  to  Socialize  with  Co-Workers,  Social 
Communication  Skills,  Task  Orientation,  Social  Skills,  Work 
Motivation,  Work  Conformance,  Personal  Presentation. 

Designed  to  'assess  fundamental  work  role 
requirements  that  are  essential  to 
achievement  and  maintenance  of  suitable 
employment'. 

"Life  Style 

Questionnair 

e" 

perseverance 

Expressive/Imaginative,  Logical/ Analytical, 
Managerial/Enterprising,  Precise/ Administrative,  Active/Concrete, 
Supportive/Social,  Risk  Taking/Uncertainty, 
Perseverance/Determination,  Self-Evaluation,  Sensitivity/Other 
Awareness,  Affiliation,  Degree  to  Which  a  Vocation  is  Associated 
with  Self  Fulfillment,  Degree  of  Certainty. 

To  provide  information  regarding 
vocational  interests  and  attitudes. 

"alternate 
uses  test" 

Fluency  of 

Ideas 

Total  score  only. 

Designed  to  represent  an  expected  factor  of 
'flexibility  of  thinking'  in  an  investigation 
of  creative  thinking.' 

Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

"The  New 
York 

Longitudinal 
Scales  Adult 
Temper  ame 
nt 

Questionnair 

e" 

perseverance 

Activity,  Rhythmicity,  Adaptability,  Threshold,  Approach, 
Distractibility,  Intensity,  Persistence,  Mood. 

Measures  temperament  in  adulthood. 

"Creativity 

Assessment 

Packet" 

Originality/Cre 

ativity 

Fluency,  Flexibility,  Originality,  Elaboration,  Titles,  Divergent 
Thinking,  Curiosity,  Imagination,  Complexity,  Risk-Taking, 
Fluency,  Flexibility,  Originality,  Elaboration,  Curiosity, 

Imagination,  Complexity,  Risk-Taking. 

To  assess  creative  potential. 

"Abbreviate 
d  Torrance 
Test  for 
Adults" 

Originality/Cre 

ativity 

Norm-Referenced  Measures  (Fluency,  Originality,  Elaboration, 
Flexibility,  Total  Scaled  Score),  Criterion-Referenced  Creativity 
Indicators  (Richness  and  Colorfulness  of  Imagery, 
Emotions/Feelings,  Future  Orientation,  Humor:  Conceptual 
Incongruity,  Provocative  Questions,  Verbal  Responses  Total, 
Openness:  Resistance  to  Premature  Closure,  Unusual 
Visualization/Different  Perspective,  Movement  and/or  Sound, 
Richness  and/or  Colorfulness  of  Imagery,  Abstractness  of  Titles, 
Articulateness  in  Telling  Story,  Combination/Synthesis  of  Two  or 
More  Figures,  Internal  Visual  Perspective,  Expressions  of  Feelings 
and  Emotions,  Fantasy,  Figural  Responses  Total),  Creativity  Index. 

To  assess  creative  thinking  ability. 

"Kirton 

Adaption- 

Innovation 

Inventory" 

Originality/Cre 

ativity 

Sufficiency  v.  Proliferation  of  Originality,  Efficiency,  Rule/Group 
Conformity,  Total. 

A  measure  of  a  person's  preference  for,  or 
style  of,  creativity,  problem  solving,  and 
decision  making. 

"Torrance 
Tests  of 
Creative 
Thinking" 

Originality/Cre 

ativity 

Verbal,  Figural,  Fluency,  Flexibility,  Originality,  Elaboration. 

To  identify  and  evaluate  creative  potential. 

Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

"Manchester 

Personality 

Questionnair 

e" 

Originality/Cre 

ativity 

Creativity,  Achievement,  Agreeableness,  Extroversion,  Resilience, 
Originality,  Rule  Consciousness,  Openness  to  Change, 

Assertiveness,  Social  Confidence,  Empathy,  Communicativeness, 
Independence,  Rationality,  Competitiveness,  Conscientiousness, 
Perfectionism,  Decisiveness,  Apprehension,  Radicalness,  Response 
Style. 

Designed  to  provide  an  occupational 
personality  test  with  a  focus  on  traits 
relevant  to  creative  and  innovative 
behavior. 

"Kaplan- 

Baycrest 

Neuropsych 

ological 

Assessment" 

Selective 

Attention, 

Concentration 

Attention/Concentration,  Immediate  Memory  Recall,  Delayed 
Memory  Recall,  Delayed  Memory  Recognition,  Spatial  Processing, 
Verbal  Fluency,  Reasoning/Conceptual  Shifting,  Total  Index. 

'Test  of  neurocognitive  functioning.' 

"California 

Computerize 

d 

Assessment 

Package" 

Selective 

Attention 

Simple  Reaction  Time,  Choice  Reaction  Time  for  Single  Digits, 
Serial  Pattern  Matching,  Lexical  Discrimination,  Visual  Selective 
Attention,  Response  Reversal  and  Rapid  Visual  Scanning,  Form 
Discrimination. 

Designed  as  a  'standardized  assessment  of 
reaction  time  and  speed  of  information 
processing.' 

"Test  of 
Attention" 

Selective 

Attention, 

Concentration 

Total  Number  of  Items  Processed  (TN),  Errors  (E),  Percentage  of 
Error  (E%),  Total  Number  of  Items  Processed  Minus  Errors  (TN- 
E),  Concentration  Performance  (CP),  Fluctuation  Rate  (FR),  and 
Skipping  Syndrome  (S-Syndrome) 

The  d2  Test  is  essentially  a  letter 
cancellation  task  intended  to  assess 
selective  attention;  it  also  measures 
sustained  attention  and  speed  of 
processing. 

"Useful 

Field  of 

View  " 

Selective 

Attention 

Central  Vision  and  Processing  Speed,  Divided  Attention,  Selective 
Attention. 

Designed  as  a  'computer-administered  and 
computer-scored  test  of  visual  attention,' 
which  may  be  used  to  help  predict  the 
degree  to  which  a  person  may  perform 
some  everyday  activities,  such  as  driving  a 
motor  vehicle,  safely. 

"Ruff  2  &  7 
Selective 
Attention 
Test" 

Selective 

Attention 

Automatic  Detection  Speed,  Automatic  Detection  Errors, 

Automatic  Detection  Accuracy,  Controlled  Search  Speed, 

Controlled  Search  Errors,  Controlled  Search  Accuracy,  Total 

Speed,  Total  Accuracy,  Speed  Difference,  Accuracy  Difference, 
Total  Difference. 

Developed  to  'measure  two  overlapping 
aspects  of  visual  attention:  sustained 
attention  and  selective  attention.' 

Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

"Test  of 

Everyday 

Attention" 

Selective 

Attention 

Map  Search,  Elevator  Counting,  Elevator  Counting  with 

Distraction,  Visual  Elevator,  Elevator  Counting  with  Reversal, 
Telephone  Search,  Telephone  Search  While  Counting,  Lottery, 

Total. 

To  measure  'selective  attention,  sustained 
attention  and  attentional  switching.' 

"Auditory 

Selective 

Attention 

Test" 

Selective 

Attention 

Total  Errors. 

'To  measure  selective  attention.' 

"Visual 

Search  and 

Attention 

Test" 

Selective 

Attention 

Left,  Right,  Total. 

Constructed  to  assess  'ability  to  scan 
accurately  and  [to]  sustain  attention  on 
each  of  four  different  visual  cancellation 
tasks'. 
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Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

Test  of 
Memory  and 
Learning 

Concentration 

10  Verbal  subtest  scores  (Memory  for  Stories,  Word  Selective 
Reminding,  Object  Recall,  Paired  Recall,  Digits  Forward,  Letters 
Forward,  Digits  Backward,  Letters  Backward,  Memory  for  Stories 
Delayed,  Word  Selective  Reminding  Delayed);  6  Nonverbal 
subtest  scores  (Facial  Memory,  Abstract  Visual  Memory,  Visual 
Sequence  Memory,  Memory  for  Location,  Visual  Selective 
Reminding,  Manual  Imitation);  3  core  composite  scores  (Verbal 
Memory  Index,  Nonverbal  Memory  Index,  Composite  Memory 
Index);  6  supplemental  composite  scores  (Verbal  Delayed  Recall 
Index,  Attention/Concentration  Index,  Sequential  Recall  Index, 

Free  Recall  Index,  Associative  Recall  Index,  Learning  Index). 

Designed  to  assess  the  'key  features  of 
memory'  and  to  'evaluate  learning  as 
reflected  in  changes  in  recall  and 
recognition  over  multiple  trials  of  various 
stimuli.' 

Wide  Range 
Assessment 
of  Memory 
and 

Learning 

Concentration 

Verbal  Memory  (Story  Memory,  Verbal  Learning,  Total),  Visual 
Memory  (Design  Memory,  Picture  Memory,  Total), 
Attention/Concentration  (Finger  Windows,  Number  Letter,  Total), 
General  Memory;  Optional  scores:  Working  Memory  (Verbal 
Working  Memory,  Symbolic  Working  Memory,  Total),  Verbal 
Recognition  (Story  Recognition,  Verbal  Learning  Recognition, 
Total),  Visual  Recognition  (Design  Recognition,  Picture  Memory 
Recognition,  Total),  General  Recognition,  Sound  Symbol, 

Sentence  Memory,  Story  Memory  Recall,  Verbal  Learning  Recall, 
Sound  Symbol  Recall. 

Designed  for  use  in  'clinical  assessments  of 
memory  including  evaluation  of  immediate 
and/or  delay  recall  as  well  as 
differentiating  between  verbal,  visual  or 
more  global  memory  deficits.' 

Learning 
and  Study 
Strategies 
Inventory 

Concentration 

Anxiety  Scale,  Attitude  Scale,  Concentration  Scale,  Information 
Processing  Scale,  Motivation  Scale,  Self-Testing  Scale,  Selecting 
Main  Ideas  Scale,  Study  Aids  Scale,  Time  Management  Scale,  Test 
Strategies  Scale. 

Designed  to  assess  'students'  awareness 
about  and  use  of  learning  and  study 
strategies  related  to  skill,  will  and  self¬ 
regulation  components  of  strategic 
learning.' 

The 

Attentional 

and 

Interpersona 

1  Style 
Inventory 

Concentration 

Attentional  (Broad  External  Awareness,  External  Distractibility, 
Conceptual/Analytical,  Internal  Distractibility,  Narrow/Focused, 
Reduced  Flexibility),  Interpersonal  (Information  Processing, 
Orientation  Toward  Rules  and  Risk/Impulse  Control,  Need  for 
Control,  Self  Esteem,  Physical  Competitiveness,  Decision  Making 
Speed,  Extroversion,  Introversion,  Expression  of  Ideas,  Expression 
of  Criticism,  Expression  of  Support,  Self-Critical). 

'Developed  to  measure  the  critical 
concentration  and  interpersonal 
determinants  of  performance.' 

Test  Name 

Attribute 

Targeted 

Test  Scores 

Test  Purpose 

START- 
Strategic 
Assessment 
of  Readiness 
for  Training 

Concentration 

Anxiety,  Attitude,  Motivation,  Concentration,  Identifying 

Important  Information,  Knowledge  Acquisition  Strategies, 
Monitoring  Learning,  Time  Management. 

Designed  to  diagnose  adult's  learning 
strengths  and  weaknesses. 

Wechsler 

Memory 

Scale 

Concentration 

Verbal  Memory  (Logical  Memory  I,  Verbal  Paired  Associates  I, 
Total),  Visual  Memory  (Figural  Memory,  Visual  Paired  Associates 

I,  Visual  Reproduction  I,  Total),  Total  General  Memory, 
Attention/Concentration  (Mental  Control,  Digit  Span,  Visual 
Memory  Span,  Total),  Delayed  Recall  (Logical  Memory  II,  Visual 
Paired  Associates  II,  Verbal  Paired  Associates  II,  Visual 
Reproduction  II,  Total),  Information  and  Orientation 

Constructed  to  assess  'memory  for  verbal 
and  figural  stimuli,  meaningful  and  abstract 
material,  and  delayed  as  well  as  immediate 
recall.' 

Non-Verbal 
Ability  Tests 

Concentration 

Matching  Shape,  Matching  Direction,  Categorization,  Picture 
Completion,  Embedded  Figures,  Figure  Formation,  Mazes, 
Sequencing,  Picture  Arrangement,  Visual  Search,  Simple  Key 

Test,  Complex  Key  Test,  Code  Tracking  I,  Code  Tracking  II, 

Visual  Recognition,  Auditory  Recognition,  Auditory  Recall,  Visual 
Recall,  Total. 

Designed  to  measure  perceptual, 
conceptual,  attention/concentration,  and 
memory  skills. 

Appendix  D.  Effects  Matrix  Summary 


Table  D-l  contains  descriptions  of  effects  matrices  that  target  one  or  more  of  the  cognitive 
abilities.  In  the  table,  N  represents  the  number  of  correlations  used  in  the  summary  effect. 
Fisher’s  Z  with  standard  error  of  Z  is  in  parentheses. 

Each  demand  group  represents  the  coding  of  an  experimental  task  that  was  correlated  with  a 
measure  of  the  personnel  attribute  by  a  standardized  test.  For  example,  there  were  four  data 
points  mined  from  studies  that  correlated  scores  on  a  standardized  test  for  flexibility  of  closure 
with  a  task  that  was  coded  as  imposing  deductive  and  inductive  demand.  In  order  to  find  the 
mean  correlation  across  those  four  data  points,  r  was  converted  to  z  (Z  takes  into  account  the 
sample  size).  Then  the  mean  z  was  computed  and  is  shown  in  the  table  with  the  standard  error  of 
z  in  parentheses.  Demand  groups  relate  to  personnel  attributes  by  z,  which  is  a  standardized 
measure  of  r,  and  essentially  represents  the  strength  of  association  between  a  score  on 
standardized  tests  that  measure  the  personnel  attribute  and  some  measure  of  performance  on  the 
experimental  task. 
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Table  D-l.  Effects  matrices  that  target  one  or  more  of  the  cognitive  abilities. 


Demand  Groups 

Measurable  Personnel  Attributes 

Analytical 

Thinking  / 

Reasoning  (18) 

Collaboration  (2) 

Creativity /Originali 

ty  (10) 

Critical  Thinking 

(36) 

Deduction  (31) 

Flexibility  of 

Closure  (64) 

Fluency  of  Ideas 

(10) 

Induction  (1) 

Inference  (7) 

Memory-Recall 

(31) 

Memory- 

Recognition  (24) 

Motivation/Perseve 

ranee  (1) 

Spatial  Rotation  (8) 

Deductive,  Inductive 

* 

* 

N=1 
.  1  (.04) 

* 

* 

N=4 

.33(.02) 

N=1 

-.01  (.04) 

* 

* 

* 

* 

* 

* 

Deductive,  Inductive, 
Flexibility  of  Closure 

N=1 
.  18(.  15) 

* 

* 

N=1 

,68(.22) 

* 

N=2 

.29(.09) 

* 

* 

* 

N=2 

-.03009) 

N=2 

■1(09) 

* 

* 

Deductive,  Inductive, 
Memorization 

N=2 

.35(.09) 

* 

* 

* 

N=2 

.35(.09) 

* 

* 

* 

N=1 
.42(.  11) 

N=1 

.37011) 

* 

* 

N=1 

-,13(.12) 

Deductive,  Inductive, 
Problem  Sensitivity 

* 

* 

* 

N=1 

1.16(.21) 

* 

* 

* 

* 

* 

* 

* 

* 

* 

Deductive,  Inductive, 
Organization  of 

Information 

* 

* 

* 

N=4 

-,07(.08) 

* 

* 

* 

* 

* 

* 

* 

* 

* 

Deductive,  Flexibility  of 
Closure 

N=1 
,44(.  15) 

* 

* 

* 

* 

N=2 
.  16(.09) 

* 

* 

* 

N=2 

.04(.09) 

N=2 
.  1 1  (.09) 

* 

* 

Deductive,  Flexibility  of 
Closure,  Problem 

Sensitivity 

* 

* 

* 

* 

* 

N=2 

.46(.09) 

* 

* 

* 

N=2 

.25(.09) 

N=2 

.23(.09) 

* 

* 

Deductive,  Flexibility  of 
Closure,  Creativity 

* 

* 

* 

N=2 
-.48(.  19) 

* 

* 

* 

* 

* 

* 

* 

* 

* 

Deductive,  Memorization, 
Organization  of 

Information 

* 

* 

* 

N=8 

.06(.06) 

N=1 
.23(.  12) 

* 

* 

* 

* 

* 

* 

* 

* 

Deductive,  Problem 
Sensitivity 

* 

* 

* 

* 

N=5 

•3(04) 

* 

* 

* 

* 

* 

* 

* 

* 

Demand  Groups 

Measurable  Personnel  Attributes 

Analytical 

Thinking  / 

Reasoning  (18) 

Collaboration  (2) 

Creativity /Originali 

ty  GO) 

Critical  Thinking 

(36) 

Deduction  (31) 

Flexibility  of 

Closure  (64) 

Fluency  of  Ideas 

GO) 

Induction  (1) 

Inference  (7) 

Memory-Recall 

(31) 

Memory - 

Recognition  (24) 

Motivation/Perseve 

ranee  (1) 

Spatial  Rotation  (8) 

Deductive,  Problem 
Sensitivity,  Organization 
of  Information 

* 

* 

* 

N=1 

.48(.06) 

* 

* 

* 

* 

* 

* 

* 

* 

* 

Deductive,  Organization 
of  Information 

* 

* 

* 

N=6 
.  14(.01) 

N=2 

-01(.l) 

* 

* 

* 

* 

* 

* 

* 

* 

Inductive,  Flexibility  of 
Closure 

N=1 
.34(.  15) 

* 

* 

N=1 

■541.1) 

* 

* 

* 

* 

* 

* 

* 

* 

* 

Inductive,  Flexibility  of 
Closure,  Problem 

Sensitivity 

* 

* 

* 

N=4 

.29(.08) 

* 

* 

* 

* 

* 

* 

* 

* 

* 

Inductive,  Flexibility  of 
Closure,  Organization  of 
Information 

* 

* 

* 

N=2 
.33(.  11) 

* 

* 

* 

* 

* 

* 

* 

* 

* 

Inductive,  Memorization 

N=4 

.47(.06) 

* 

* 

* 

N=4 

.34(.06) 

* 

* 

* 

N=3 

.35(37) 

N=3 

.49(37) 

* 

* 

N=3 

-31(37) 

Inductive,  Fluency  of 

Ideas,  Creativity 

* 

N=1 

.941.19) 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

Collaboration 

* 

* 

* 

N=1 

-.3(35) 

* 

* 

* 

* 

* 

* 

* 

N=1 

.62(33) 

* 

Collaboration, 
Memorization,  Problem 
Sensitivity 

* 

* 

* 

N=1 
.  19(.09) 

* 

* 

* 

* 

* 

* 

* 

* 

* 

Flexibility  of  Closure 

N=1 
.41  (.  15) 

* 

N=7 
.  1  (.02) 

* 

* 

N=24 

.22(31) 

N=7 

.04(32) 

* 

* 

* 

* 

* 

* 

Demand  Groups 

Measurable  Personnel  Attributes 

Analytical 

Thinking  / 

Reasoning  (18) 

Collaboration  (2) 

Creativity /Originali 

ty  GO) 

Critical  Thinking 

(36) 

Deduction  (31) 

Flexibility  of 

Closure  (64) 

Fluency  of  Ideas 

GO) 

Induction  (1) 

Inference  (7) 

Memory-Recall 

(31) 

Memory - 

Recognition  (24) 

Motivation/Perseve 

ranee  (1) 

Spatial  Rotation  (8) 

Flexibility  of  Closure, 
Memorization 

* 

* 

* 

* 

* 

N=2 

,3(.09) 

* 

* 

* 

N=4 

.22(.06) 

N=3 

.22(.07) 

* 

* 

Flexibility  of  Closure, 
Memorization, 

Organization  of 

Information 

N=2 

.55C09) 

* 

* 

N=2 
.01  (.08) 

N=5 

,15(.06) 

* 

* 

* 

* 

* 

* 

* 

N=1 

11(12) 

Flexibility  of  Closure, 
Problem  Sensitivity, 
Perseverance 

* 

* 

* 

N=2 

,01(.09) 

* 

* 

* 

* 

* 

* 

* 

* 

* 

Flexibility  of  Closure, 
Fluency  of  Ideas 

* 

* 

* 

* 

* 

N=2 
.  1 1  (.09) 

* 

* 

* 

N=2 

-,03(.09) 

N=2 

.02(.09) 

* 

* 

Flexibility  of  Closure, 
Fluency  of  Ideas, 

Creativity 

N=1 

,32(,15) 

* 

* 

* 

* 

N=2 

,52(.09) 

* 

* 

* 

N=2 

.21(.09) 

N=2 
.41  (.09) 

* 

* 

Flexibility  of  Closure, 
Timesharing 

* 

* 

N=1 

,08(.04) 

* 

* 

N=4 

,21(.02) 

N=1 

,23(.04) 

* 

* 

* 

* 

* 

* 

Memorization 

N=2 

.44(.09) 

* 

* 

* 

N=2 

,37(.09) 

* 

* 

* 

N=1 
•47(.  11) 

N=1 
,54(.l  1) 

* 

* 

N=1 

-.09(.  12) 

Memorization, 

Organization  of 

Information 

N=3 

.28(.07) 

* 

* 

* 

N=3 

,31(.07) 

N=6 

.25(.05) 

* 

* 

N=2 

.38(.08) 

N=6 

,29(.05) 

N=5 

.17(.06) 

* 

N=2 
.  17(.09) 

Memorization, 

Organization  of 

Information,  Selective 
Attention 

* 

* 

* 

* 

N=4 

.591.09) 

* 

* 

* 

* 

* 

* 

* 

* 

Demand  Groups 

Measurable  Personnel  Attributes 

Analytical 

Thinking  / 

Reasoning  (18) 

Collaboration  (2) 

Creativity /Originali 

ty  GO) 

Critical  Thinking 

(36) 

Deduction  (31) 

Flexibility  of 

Closure  (64) 

Fluency  of  Ideas 

GO) 

Induction  ( 1 ) 

Inference  (7) 

Memory-Recall 

(31) 

Memory - 

Recognition  (24) 

Motivation/Perseve 

ranee  (1) 

Spatial  Rotation  (8) 

Memorization, 
Timesharing,  Selective 
Attention 

* 

* 

* 

* 

* 

N=2 

,36(,09) 

* 

* 

* 

N=2 

,23(,09) 

N=2 
.21  (.09) 

* 

* 

Memorization,  Creativity 

* 

* 

* 

* 

* 

N=2 

,46(,09) 

* 

* 

* 

N=2 

,15(,09) 

N=2 
.25 (.09) 

* 

* 

Problem  Sensitivity 

* 

* 

* 

* 

N=1 

-,41(.  16) 

* 

* 

* 

* 

* 

* 

* 

* 

Problem  Sensitivity, 
Organization  of 
Information,  Timesharing 

* 

* 

* 

* 

* 

N=2 

,18(,09) 

* 

* 

* 

N=2 

,06(.09) 

N=2 

,2(,09) 

* 

* 

Organization  of 
Information,  Fluency  of 
Ideas,  Creativity 

* 

* 

* 

* 

N=1 
,04(.  13) 

* 

* 

N=1 
,01(.  13) 

* 

* 

* 

* 

* 

Organization  of 
Information,  Timesharing 

* 

* 

* 

* 

N=1 

■3(.25) 

* 

* 

* 

* 

* 

* 

* 

* 

Perseverance 

* 

N=1 

,62(,03) 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

* 

Fluency  of  Ideas 

* 

* 

N=1 

,03(,04) 

* 

* 

N=4 

,04(,02) 

* 

* 

* 

* 

* 

* 

* 

Fluency  of  Ideas, 

Creativity 

* 

* 

* 

* 

* 

N=4 

,12(,02) 

N=1 

,03(,04) 

* 

* 

* 

* 

* 

* 

Intentionally  lelt  blank. 
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Appendix  E.  Detailed  Task  Data  Used  for  Model  Development 


Table  E-l  contains  the  ratings  given  by  the  four  subject  matter  experts.  In  some  cases  an  ability 
was  not  rated,  but  that  should  not  infer  that  it  is  not  critical  overall. 

SMEs  provided  information  about  the  subtasks  conducted  during  the  course  of  performing  an 
Evaluate  the  Threat  task.  The  time  duration  taken  to  perform  these  subtasks,  along  with  their 
frequency  of  occurrence,  is  provided  in  table  E-2.  A  distinction  was  made  to  differentiate  full- 
spectrum  operations  from  stability  and  support  operations  since  some  of  the  subtasks  involve  a 
significantly  different  amount  of  effort  depending  on  the  type  of  operating  being  worked. 
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Table  E-l.  Subject  matter  expert  ratings  of  ability  importance  to  task. 


Activity  (or  Task):  Evaluate  the 
threat 

Ability  (see  descriptions) 

Needed  to  perform  the  activity: 

1  =  a  minimum  amount  is  needed 

4  =  a  moderate  amount  is  needed 

7  =  a  great  amount  is  needed 

1. 

Memor 

ization 

2. 

Selectiv 

e 

Attentio 

n 

3. 

Time 

Sharin 

g 

4. 

Deducti 

ve 

Reasoni 

ng 

5. 

Inductiv 

e 

Reasoni 

ng 

6. 

Problem 

Sensitivi 

ty 

7. 

Flexibili 
ty  of 
Closure 

8. 

Fluency 
of  Ideas 

9. 

Originalit 

y 

10. 

Persever 

ance 

20/30/40:  Prioritization  of  tasks 

6 

7 

4 

4 

7 

6 

6 

5 

6 

5 

4 

6 

6 

4 

4 

10:  Identify  METT-TC  effects  on  threat 

4 

4 

5 

6 

5 

3 

2 

1 

1 

2 

forces 

40:  Verify  METT-TC  effects  on  threat  forces 

3 

4 

5 

5 

5 

3 

2 

1 

1 

1 

10;  Vpdate  COP  with  current  enemy 

5 

4 

5 

5 

6 

6 

6 

6 

3 

5 

composition 

40:  Validate  COP  with  current  enemy 

5 

4 

5 

5 

5 

5 

6 

6 

4 

6 

composition 

10:  Develop  Link  Diagrams,  Association 

3 

4 

6 

6 

6 

6 

6 

5 

5 

5 

Matrices,  Activity  Matrices,  and  TEC 

30:  Verify  Link  Diagrams,  Association 

6 

5 

5 

6 

5 

5 

4 

Matrices,  Activity  Matrices,  and  TEC 

6 

6 

6 

6 

5 

5 

5 

4 

5 

6 

4 

3 

4 

6 

6 

6 

6 

6 

5 

5 

5 

40:  Analyze  Link  Diagrams,  Association 

7 

6 

7 

7 

7 

7 

7 

6 

6 

7 

Matrices,  Activity  Matrices,  and  TEC 

20:  Incorporate  the  elements  (battlefield 

5 

6 

4 

geography  and  environment  conditions)  by 

5 

6 

6 

7 

understanding  the  elements  of  the 

f. 

s 

f. 

environment  and  how  they  affect  the  battle. 

4 

4 

5 

4 

4 

4 

4 

2 

1 

1 

40:  Verify  the  elements  have  been 

1 

2 

2 

3 

1 

3 

1 

1 

2 

incorporated  into  the  threat  model. 

10:  Identify  gaps  in  intelligence  holdings 

7 

6 

7 

5 

6 

6 

7 

Activity  (or  Task):  Evaluate  the 
threat 

Ability  (see  descriptions) 

Needed  to  perform  the  activity: 

1  =  a  minimum  amount  is  needed 

4  =  a  moderate  amount  is  needed 

7  =  a  great  amount  is  needed 

1. 

Memor 

ization 

2. 

Selectiv 

e 

Attentio 

n 

3. 

Time 

Sharin 

g 

4. 

Deducti 

ve 

Reasoni 

ng 

5. 

Inductiv 

e 

Reasoni 

ng 

6. 

Problem 

Sensitivi 

ty 

7. 

Flexibili 
ty  of 
Closure 

8. 

Fluency 
of  Ideas 

9. 

Originalit 

y 

10. 

Persever 

ance 

40:  Verify  gaps  in  intelligence  holdings 

7 

6 

7 

5 

6 

6 

7 

10:  Identify  threat  capabilities 

7 

7 

7 

6 

7 

7 

6 

10:  Draft  threat  assessments  to  develop 

7 

6 

7 

6 

7 

7 

6 

realistic  threat  models 

Create  threat  assessments  to  develop  realistic 

5 

6 

7 

threat  models 

5 

6 

6 

7 

3 

4 

6 

5 

7 

6 

7 

7 

7 

7 

6 

6 

7 

Develop  enemy  OB/structure 

7 

6 

6 

6 

5 

7 

6 

5 

7 

7 

6 

7 

6 

7 

7 

7 

40:  Verify  enemy  OB/structure 

7 

6 

7 

6 

7 

7 

7 

Develop  threat  capabilities  statement  in  Full 

7 

6 

6 

6 

4 

Spectrum  is  focused  on  units 

6 

6 

6 

6 

4 

6 

5 

7 

6 

7 

7 

7 

7 

6 

7 

Develop  threat  capabilities  statement  in 

7 

6 

6 

6 

Stability  and  Support  is  focused  on 

6 

6 

6 

6 

7 

individual  or  small  group  level  sustainment 

capabilities 

J 

J 

O 

o 

7 

6 

7 

6 

7 

7 

7 

40:  Verify  threat  capabilities  statement 

7 

6 

7 

6 

7 

7 

7 

Identify  relevant  databases 

7 

5 

5 

6 

6 

5 

4 

7 

7 

7 

4 

7 

4 

7 

5 

5 

7 

Activity  (or  Task):  Evaluate  the 
threat 

Ability  (see  descriptions) 

Needed  to  perform  the  activity: 

1  =  a  minimum  amount  is  needed 

4  =  a  moderate  amount  is  needed 

7  =  a  great  amount  is  needed 

1. 

Memor 

ization 

2. 

Selectiv 

e 

Attentio 

n 

3. 

Time 

Sharin 

g 

4. 

Deducti 

ve 

Reasoni 

ng 

5. 

Inductiv 

e 

Reasoni 

ng 

6. 

Problem 

Sensitivi 

ty 

7. 

Flexibili 
ty  of 
Closure 

8. 

Fluency 
of  Ideas 

9. 

Originalit 

y 

10. 

Persever 

ance 

Analyze  intelligence  holding  IOT  identify 

7 

6 

6 

5 

existing  or  emerging  enemy  TTPs 

6 

6 

7 

6 

6 

5 

4 

7 

7 

7 

7 

7 

7 

7 

6 

6 

7 

Develop  initial  ECO  As  based  upon 

5 

6 

6 

6 

6 

indicators  and  historical  TTPs 

6 

7 

6 

5 

6 

6 

5 

5 

7 

7 

7 

7 

7 

7 

7 

6 

5 

7 

40:  Verify  initial  ECOAs  based  upon 

7 

7 

7 

7 

7 

7 

7 

6 

3 

7 

indicators  and  historical  TTPs 

Depict  the  composition  and  array  of  enemy 

6 

6 

6 

6 

network  in  an  AO,  AI,  and  AOI  based  on 

5 

6 

6 

operational  variables  (PMESII-PT  ASCOPE) 

3 

6 

2 

6 

and  TTPs. 

7 

5 

7 

7 

7 

7 

7 

4 

4 

7 

20:  Develop  RFIs 

7 

6 

5 

5 

5 

6 

6 

5 

6 

5 

5 

4 

4 

4 

3 

4 

7 

30:  Compile  RFIs 

6 

4 

4 

6 

3 

3 

7 

Develop  HVI/HVT/HPTs 

6 

4 

6 

6 

6 

6 

5 

6 

6 

5 

5 

6 

7 

5 

7 

7 

7 

7 

7 

5 

5 

7 

Validate  HVI/HVT/HPTs 

7 

7 

7 

7 

7 

7 

7 

7 

3 

7 

Table  E-2.  Evaluate  the  Threat  subtask  occurrences. 


oo 


Task 


Activity  (or  Task) 


Task  Time 


Performance  Levels 


Oper 


Task  Duration 


Frequency  of  Occurrence 


Performance  is  considered  to  be 
either: 

1)  satisfactory  or  unsatisfactory,  or 

2)  on  a  scale  of  quality  such  as  one 
that  ranges  from  inadequate, 
through  poor,  OK,  good,  excellent? 


SL  10 


SL  20 


SL  30 


SL  40 


SL  10 


SL  20 


SL  30 


SL40 


Prioritization  of  tasks 


Sporadically  all  the  time,  as  needed. 
Seconds  to  a  few  minutes. 


inadequate,  through  poor,  OK,  good, 
excellent 


all 


10  sec  -  3-5  mins 


Sporadically  all  the 
time 


Depict  the  composition  and  array  of 
enemy  network  in  an  AO,  AI,  and  AO  I 
based  on  operational  variables 
(PMESII-PT  ASCOPE)  andTTPs. 


FSO  is  quick  and  easy.  Characteristics 
of  enemy  are  known.  On  a  12-hr  shift 
it  might  take  1-2  hr,  or  more  detail  may 
need  more  based  on  Cmdr  needs. 
Company  may  take  3-4  hrs.  Battalion  1- 
2  hrs  due  to  capture  two  echelons 
below  you. 


SSO  is  ongoing  each  time  a  new  person, 
weapon  shows  up.  As  time  goes  by 
you  develop  a  knowledge  of  the 
community  fills  in. 


Creating  and  presenting  products  that 
meet  Cmdrs  rqts  and  needs.  Accuracy 
and  relevant  with  backup.  This  is  the 
major  presentation  to  the  Cmdr  every 
12  hrs.  About  .25  %  of  time  there  is  a 
lack  info/support 


FSO 


SSO 


1-2  hrs 


10-20  mins 


12-24  hrs 


Develop  RFIs 


Fill  in  a  form,  identify  gaps  (this  can 
take  time  2-3  hrs,  to  coalesce),  request 
in  a  manner  that  is  specific  and 
answerable.  As  the  gaps  get  smaller 
the  time  can  get  down  to  ~  1  hr. 


Gap  is  found  or  not  found.  Request  is 
not  well  formed.  Gap  is  not  adequately 
investigated  or  understood. 


all 


Update  COP  with  current  enemy 
composition 


all 


Develop  HVI/HVT/HPTs.  This  is  part 
of  Operations  Planning.  Done  mainly 
at  night.  Prioritize  the  weapons.  This 
is  a  preparatory  task  that  can  take 
many  hours.  Prioritization. 
Conservation  of  assets  -  take  out  the 
radar  vs  the  six  weapon  systems 
supported  by  it 


FSO  are  known  capabilities 


FSO 


SSO  are  individual  and  weapon  based, 
ongoing.  Could  take  weeks  or  month 
overall. 


SSO 


1-3  hrs 


Verify  gaps  in  intelligence  holdings 


all 


Identify  relevant  databases 


Really  more  of  a  knowledge  base 
gained  over  the  years.  Minutes  to  .5  to 
2-3  hrs  (maybe  10%  of  the  time) 


Poor,  good,  very  good,  great.  Gather  a 
set  of  databases  or  even  create  their 


all 


3-5  min  to  2-3  hrs 


10% 


00 


Task 

Activity  (or  Task) 

Task  Time 

Performance  Levels 

Oper 

ation 

Task  Duration 

Frequency  of  Occurrence 

Performance  is  considered  to  be 

either: 

1)  satisfactory  or  unsatisfactory,  or 

2)  on  a  scale  of  quality  such  as  one 
that  ranges  from  inadequate, 
through  poor,  OK,  good,  excellent? 

SL 10 

SL  20 

SL  30 

SL  40 

SL  10 

SL  20 

SL  30 

SL  40 

8 

Verify  METT-TC  effects  on  threat 
forces 

all 

10-20 

min 

9 

Draft  threat  assessments  to  develop 
realistic  threat  models 

all 

1-2 

hrs 

10 

Develop  enemy  OB/structure  - 
indicate  a  level  of  confidence  in 
information  you  have  gathered 

A  given  in  FSO  mode  with  known 
enemy  (eg,  Korea),  1-2  hrs 

FSO 

1-2  hrs 

every  1-3  mo 

SSO  is  ongoing,  1-2  min  or  more  per 
event,  max  of  2-3  hrs  (very  infrequent) 

SSO 

1-2  min  to  2-3  hrs 

ongoing 

11 

Analyze  intelligence  holding  IOT 
identify  existing  or  emerging  enemy 
TTPs 

FSO  is  <  lhr.  This  occurs  rarely,  1- 
2/yr 

Success  or  fail  to  perform  due  to  lack  of 
investigation  or  persistence.  With  IA 
experience  you  get  better 

FSO 

20-60  min 

1-2  per  year 

SSO  is  ongoing  when  you  get  a  new 
piece  of  info.  You  must  do 
investigation  on  each  new  individual 
or  system  (weapon)  that  shows  up. 

SSO 

2-8  hrs 

ongoing 

12 

Develop  initial  ECOAs  based  upon 
indicators  and  historical  TTPs 

FSO  it  takes  a  couple  days  done  prior 
to  deployment.  Develop  about  3  COAs. 
Cmdr's  discretion  and  time 
constrained. 

Poor  performance  is  due  to  being 
sloppy.  Not  taking  something  into 
consideration.  Dropping  the  ball  too 
many  times.  An  attitude  problem. 

FSO 

12-36  hrs 

1-2  per  year 

SSO  depends  on  the  enemy.  Review  of 
past  30  days  to  update  SA.  1-2  days  of 
focused  effort.  Become  an  SME  on 
them  for  their  very  recent  history. 

Redo  this  every  72  hrs  or  so. 

SSO 

12-24  hrs 

60  -  84  hrs 

13 

Develop  threat  capabilities  statement 
in  Full  Spectrum  is  focused  on  units  - 
a  quick  analysis  and  writing  one 
paragraph 

Fairly  easy,  about  an  hour,  dealing 
with  knowns 

20/30  level  should  easily  get  this  99% 
always 

FSO 

~lhr 

every  day 

13 

Develop  threat  capabilities  statement 
in  Stability  and  Support  is  focused  on 
individual  or  small  group  level 
sustainment  capabilities  -  requires 
much  more  analysis  and  is  ongoing. 

Use  of  FMs,  TTPs,  methods,  checklists 
to  guide  performance. 

Ongoing,  time  to  investigate 
individuals  or  weapon  system,  hrs  to 
days,  you  run  across  a  higher 
percentage  of  surprises  which  need 
investigation 

Mistakes  are  made  when  the  checklists 
are  not  used  when  you  are  not  sure  of 
the  method.  Use  of  Initiative  is  what 
good  analysts  does  if  method  is  not 
available  or  known. 

SSO 

2-8  hrs 

ongoing 

NO.  OF 

COPIES  ORGANIZATION 


1  DEFENSE  TECHNICAL 
(PDF  INFORMATION  CTR 
only)  DTIC  OCA 

8725  JOHN  J  KINGMAN  RD 
STE  0944 

FORT  BELVOIR  VA  22060-6218 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
IMNE  ALC  HRR 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  CIO  LL 
2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 

1  DIRECTOR 

US  ARMY  RESEARCH  LAB 
RDRL  D 

2800  POWDER  MILL  RD 
ADELPHI  MD  20783-1 197 
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1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  C  A  DAVISON 

320  MANSCEN  LOOP  STE  1 15 
FORT  LEONARD  WOOD  MO  65473 

2  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DI 

T  DAVIS 

J  HANSBERGER 

BLDG  5400  RM  C242 

REDSTONE  ARSENAL  AL  35898-7290 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRS  EA  DR  V  J  RICE 
BLDG  4011  RM  217 
1750  GREELEY  RD 

FORT  SAM  HOUSTON  TX  78234-5002 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  DG  K  GUNN 
BLDG  333 

PICATINNY  ARSENAL  NJ  07806-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

ARMC  FIELD  ELEMENT 
RDRL  HRM  CH  C  BURNS 
THIRD  AVE  BLDG  1467B  RM  336 
FORT  KNOX  KY  40121 

1  ARMY  RSCH  LABORATORY  -  HRED 

AWC  FIELD  ELEMENT 
RDRL  HRM  DI  D  DURBIN 
BLDG  4506  (DCD)  RM  107 
FORT  RUCKER  AL  36362-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  CK  J  REINHART 
10125  KINGMAN  RD  BLDG  317 
FORT  BELVOIR  VA  22060-5828 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HRM  AY  M  BARNES 
2520  HEALY  AVE 
STE  1172  BLDG  51005 
FORT  HUACHUCA  AZ  85613-7069 

1  ARMY  RSCH  LABORATORY  -  HRED 

RDRL  HR  MP  D  UNGV ARSKY 
POPE  HALL  BLDG  470 
BCBL  806  HARRISON  DR 
FORT  LEAVENWORTH  KS  66027-2302 


1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  DQ  MR  FLETCHER 
NATICK  SOLDIER  CTR 
AMSRD  NSC  WS  E  BLDG  3  RM  343 
NATICK  MA  01760-5020 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  J  CHEN 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826-3276 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AT  C  KORTENHAUS 
12350  RESEARCH  PKWY 
ORLANDO  FL  32826 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AS  C  MANASCO 
SIGNAL  TOWERS 
BLDG  29808A  RM  303A 
FORT  GORDON  GA  30905-5233 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CU 
6501  E  11  MILE  RD  MS  284 
BLDG200A  2ND  FL  RM2104 
WARREN  MI  48397-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
FIRES  CTR  OF  EXCELLENCE 
FIELD  ELEMENT 
RDRL  HRM  AF  C  HERNANDEZ 
3040  NW  AUSTIN  RD  RM  221 
FORT  SILL  OK  73503-9043 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  AV  S  MIDDLEBROOKS 
91012  STATION  AVE 
FORT  HOOD  TX  76544-5073 

1  ARMY  RSCH  LABORATORY  -  HRED 
RDRL  HRM  CN  R  SPENCER 
DCSFDI  HF 

HQUSASOC  BLDG  E2929 
FORT  BRAGG  NC  28310-5000 

1  ARMY  RSCH  LABORATORY  -  HRED 
HUMAN  RSRCH  AND  ENGRNG 
DIRCTRT  MCOE  FIELD  ELEMENT 
RDRL  HRM  DW  E  REDDEN 
6450  WAY  ST 
BLDG  2839  RM  310 
FORT  BENNING  GA  31905-5400 
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1  ARMY  G1 
(CD  DAPE  MR  B  KNAPP 
only)  300  ARMY  PENTAGON  RM  2C489 
WASHINGTON  DC  20310-0300 


ABERDEEN  PROVING  GROUND 

5  DIR  USARL 
RDRL  HR 
L  ALLENDER 
T  LETOWSKI 
RDRL  HRM 

P  S  A V AGE-KNEPS  HIELD 
RDRL  HRM  A 
D  HARRAH 
RDRL  HRS  D 
B  AMREIN 
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